[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:55:57 dist-test-slave-2g0f krb5kdc[2338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:55:57 dist-test-slave-2g0f krb5kdc[2338](info): set up 2 sockets
Sep 25 21:55:57 dist-test-slave-2g0f krb5kdc[2338](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 21:55:58.367663 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:55:58 dist-test-slave-2g0f krb5kdc[2338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:55:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:55:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:55:58.487740 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:35847
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:35847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:55:58.566406 2351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:55:58.566624 2351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:55:58.566655 2351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:55:58.567849 2351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:55:58.567903 2351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:55:58.567921 2351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:55:58.567932 2351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:55:58.567943 2351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:55:58.569407 2351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:35847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:35847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:55:58.569605 2351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:55:58.569801 2351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:55:58.575349 2356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:55:58.575335 2357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:55:58.575397 2351 server_base.cc:1034] running on GCE node
W20260925 21:55:58.575335 2359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:55:58.575814 2351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:55:58.576151 2351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:55:58.577306 2351 hybrid_clock.cc:648] HybridClock initialized: now 1790373358577286 us; error 35 us; skew 500 ppm
Sep 25 21:55:58 dist-test-slave-2g0f krb5kdc[2338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373358, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:55:58.579778 2351 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:55:58.580238 2351 webserver.cc:460] Webserver started at http://127.2.70.254:40131/ using document root <none> and password file <none>
I20260925 21:55:58.580464 2351 fs_manager.cc:359] Metadata directory not provided
I20260925 21:55:58.580509 2351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:55:58.580629 2351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:55:58.581776 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "fc39d926266c4487b103af00d866735a"
format_stamp: "Formatted at 2026-09-25 21:55:58 on dist-test-slave-2g0f"
server_key: "68a9432b08d992efb0a5711565372c01"
server_key_iv: "b611bc31f3e2b2f4a9ffe89771240e75"
server_key_version: "encryptionkey@0"
I20260925 21:55:58.582126 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "fc39d926266c4487b103af00d866735a"
format_stamp: "Formatted at 2026-09-25 21:55:58 on dist-test-slave-2g0f"
server_key: "68a9432b08d992efb0a5711565372c01"
server_key_iv: "b611bc31f3e2b2f4a9ffe89771240e75"
server_key_version: "encryptionkey@0"
I20260925 21:55:58.583513 2351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 21:55:58.584385 2366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:55:58.584657 2351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:55:58.584748 2351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "fc39d926266c4487b103af00d866735a"
format_stamp: "Formatted at 2026-09-25 21:55:58 on dist-test-slave-2g0f"
server_key: "68a9432b08d992efb0a5711565372c01"
server_key_iv: "b611bc31f3e2b2f4a9ffe89771240e75"
server_key_version: "encryptionkey@0"
I20260925 21:55:58.584824 2351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:55:58.594667 2351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:55:58.596727 2351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:55:58.596860 2351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:55:58.600603 2351 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:35847
I20260925 21:55:58.600641 2418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:35847 every 8 connection(s)
I20260925 21:55:58.600953 2351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:55:58.601794 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:55:58.602250 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 2351
I20260925 21:55:58.602330 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:55:58.602998 2331 external_mini_cluster.cc:1428] Setting key 4283690122f3b8c59a8f5b3f4f1d062b
I20260925 21:55:58.604483 2419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a: Bootstrap starting.
I20260925 21:55:58.605008 2419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a: Neither blocks nor log segments found. Creating new log.
I20260925 21:55:58.605283 2419 log.cc:824] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a: Log is configured to *not* fsync() on all Append() calls
I20260925 21:55:58.605959 2419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a: No bootstrap required, opened a new log
I20260925 21:55:58.607565 2419 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc39d926266c4487b103af00d866735a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35847 } }
I20260925 21:55:58.607710 2419 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:55:58.607748 2419 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc39d926266c4487b103af00d866735a, State: Initialized, Role: FOLLOWER
I20260925 21:55:58.607934 2419 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc39d926266c4487b103af00d866735a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35847 } }
I20260925 21:55:58.608026 2419 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:55:58.608062 2419 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:55:58.608114 2419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:55:58.608691 2419 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc39d926266c4487b103af00d866735a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35847 } }
I20260925 21:55:58.608790 2419 leader_election.cc:302] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc39d926266c4487b103af00d866735a; no voters:
I20260925 21:55:58.608961 2419 leader_election.cc:288] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:55:58.609002 2424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:55:58.609169 2424 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [term 1 LEADER]: Becoming Leader. State: Replica: fc39d926266c4487b103af00d866735a, State: Running, Role: LEADER
I20260925 21:55:58.609309 2424 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc39d926266c4487b103af00d866735a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35847 } }
I20260925 21:55:58.609340 2419 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:55:58.609647 2424 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc39d926266c4487b103af00d866735a. Latest consensus state: current_term: 1 leader_uuid: "fc39d926266c4487b103af00d866735a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc39d926266c4487b103af00d866735a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35847 } } }
I20260925 21:55:58.609724 2424 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [sys.catalog]: This master's current role is: LEADER
I20260925 21:55:58.609894 2426 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc39d926266c4487b103af00d866735a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc39d926266c4487b103af00d866735a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35847 } } }
I20260925 21:55:58.610026 2426 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a [sys.catalog]: This master's current role is: LEADER
I20260925 21:55:58.610272 2429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:55:58.610798 2429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:55:58.612457 2429 catalog_manager.cc:1347] Generated new cluster ID: fb8dc3c2c4c944e0b4f65498c82c6d9d
I20260925 21:55:58.612497 2429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 21:55:58 dist-test-slave-2g0f krb5kdc[2338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:55:58.620913 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:55:58.603814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.2.70.254:35847)
0925 21:55:58.603987 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 21:55:58.603995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 21:55:58.604268 (+ 273us) server_negotiation.cc:407] Connection header received
0925 21:55:58.606326 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:55:58.606332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:55:58.606381 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:55:58.606504 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:55:58.608024 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:55:58.609416 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:55:58.611055 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:55:58.611882 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:55:58.612007 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:55:58.616107 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:55:58.616111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:55:58.616112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:55:58.616124 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:55:58.618663 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:55:58.619150 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:55:58.619154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:55:58.619154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:55:58.619270 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:55:58.619523 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:55:58.619523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:55:58.619524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:55:58.619913 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:55:58.620075 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 21:55:58.620355 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 21:55:58.620497 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260925 21:55:58.628118 2429 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:55:58.628614 2429 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:55:58.634598 2429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc39d926266c4487b103af00d866735a: Generated new TSK 0
I20260925 21:55:58.634755 2429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:55:58.659999 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35847
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:55:58.740420 2445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:55:58.740644 2445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:55:58.740669 2445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:55:58.741868 2445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:55:58.741914 2445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:55:58.741940 2445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:55:58.743441 2445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35847
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:55:58.743667 2445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:55:58.743927 2445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:55:58.748574 2450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:55:58.748580 2451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:55:58.748788 2453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:55:58.748783 2445 server_base.cc:1034] running on GCE node
I20260925 21:55:58.749037 2445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:55:58.749266 2445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:55:58.750417 2445 hybrid_clock.cc:648] HybridClock initialized: now 1790373358750404 us; error 26 us; skew 500 ppm
Sep 25 21:55:58 dist-test-slave-2g0f krb5kdc[2338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373358, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:55:58.752856 2445 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:55:58.753261 2445 webserver.cc:460] Webserver started at http://127.2.70.193:43039/ using document root <none> and password file <none>
I20260925 21:55:58.753458 2445 fs_manager.cc:359] Metadata directory not provided
I20260925 21:55:58.753504 2445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:55:58.753623 2445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:55:58.754673 2445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "3ead30d6c3d14310986a7b45c1bf01b8"
format_stamp: "Formatted at 2026-09-25 21:55:58 on dist-test-slave-2g0f"
server_key: "c7dd3043ab26ccd7fc83950b4f6fb405"
server_key_iv: "c1b5e13cb870b8375ac58d84312c9c56"
server_key_version: "encryptionkey@0"
I20260925 21:55:58.755008 2445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "3ead30d6c3d14310986a7b45c1bf01b8"
format_stamp: "Formatted at 2026-09-25 21:55:58 on dist-test-slave-2g0f"
server_key: "c7dd3043ab26ccd7fc83950b4f6fb405"
server_key_iv: "c1b5e13cb870b8375ac58d84312c9c56"
server_key_version: "encryptionkey@0"
I20260925 21:55:58.756266 2445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 21:55:58.757068 2460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:55:58.757236 2445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:55:58.757314 2445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "3ead30d6c3d14310986a7b45c1bf01b8"
format_stamp: "Formatted at 2026-09-25 21:55:58 on dist-test-slave-2g0f"
server_key: "c7dd3043ab26ccd7fc83950b4f6fb405"
server_key_iv: "c1b5e13cb870b8375ac58d84312c9c56"
server_key_version: "encryptionkey@0"
I20260925 21:55:58.757400 2445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:55:58.773105 2445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:55:58.775127 2445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:55:58.775273 2445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:55:58.775525 2445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:55:58.775815 2445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:55:58.775877 2445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:55:58.775909 2445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:55:58.775934 2445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:55:58.781566 2445 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:33825
I20260925 21:55:58.781610 2573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:33825 every 8 connection(s)
I20260925 21:55:58.781925 2445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:55:58.784853 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 2445
I20260925 21:55:58.784933 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:55:58.785068 2331 external_mini_cluster.cc:1428] Setting key edf71a69810ce6fdd6a9bf2165459e2f
Sep 25 21:55:58 dist-test-slave-2g0f krb5kdc[2338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373358, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:55:58.791577 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:55:58.782643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:43025 (local address 127.2.70.254:35847)
0925 21:55:58.782704 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 21:55:58.782706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:55:58.782889 (+ 183us) server_negotiation.cc:407] Connection header received
0925 21:55:58.783436 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:55:58.783437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:55:58.783451 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:55:58.783502 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:55:58.784518 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:55:58.785122 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:55:58.786208 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:55:58.786437 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:55:58.786492 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:55:58.788739 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:55:58.788740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:55:58.788741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:55:58.788749 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:55:58.790204 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:55:58.790585 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:55:58.790586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:55:58.790587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:55:58.790650 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:55:58.790855 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:55:58.790856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:55:58.790856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:55:58.790997 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:55:58.791070 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 21:55:58.791363 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 21:55:58.791455 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 21:55:58.791682 2576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:55:58.782604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35847 (local address 127.2.70.193:43025)
0925 21:55:58.782772 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 21:55:58.782810 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 21:55:58.783304 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:55:58.783579 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:55:58.783584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:55:58.784211 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:55:58.784420 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:55:58.784425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:55:58.785208 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:55:58.785209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:55:58.786130 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:55:58.786133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:55:58.786507 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:55:58.786507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:55:58.786631 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:55:58.787107 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:55:58.787115 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:55:58.788661 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:55:58.790285 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:55:58.790291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:55:58.790293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:55:58.790516 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:55:58.790708 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:55:58.790709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:55:58.790709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:55:58.790803 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:55:58.791089 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:55:58.791096 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:55:58.791221 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 21:55:58.791330 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 21:55:58.791480 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260925 21:55:58.792398 2574 heartbeater.cc:347] Connected to a master server at 127.2.70.254:35847
I20260925 21:55:58.792491 2574 heartbeater.cc:464] Registering TS with master...
I20260925 21:55:58.792666 2574 heartbeater.cc:511] Master 127.2.70.254:35847 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
I20260925 21:55:58.793226 2383 ts_manager.cc:194] Registered new tserver with Master: 3ead30d6c3d14310986a7b45c1bf01b8 (127.2.70.193:33825)
I20260925 21:55:58.793825 2383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:43025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:55:58.810088 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35847
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:55:58.891574 2579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:55:58.891806 2579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:55:58.891860 2579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:55:58.893061 2579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:55:58.893111 2579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:55:58.893141 2579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:55:58.894675 2579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35847
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:55:58.894886 2579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:55:58.895069 2579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:55:58.900087 2587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:55:58.900087 2584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:55:58.900192 2585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:55:58.900243 2579 server_base.cc:1034] running on GCE node
I20260925 21:55:58.900556 2579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:55:58.900794 2579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:55:58.901990 2579 hybrid_clock.cc:648] HybridClock initialized: now 1790373358901975 us; error 28 us; skew 500 ppm
Sep 25 21:55:58 dist-test-slave-2g0f krb5kdc[2338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373358, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:55:58.904598 2579 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:55:58.905114 2579 webserver.cc:460] Webserver started at http://127.2.70.194:40801/ using document root <none> and password file <none>
I20260925 21:55:58.905325 2579 fs_manager.cc:359] Metadata directory not provided
I20260925 21:55:58.905375 2579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:55:58.905516 2579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:55:58.906651 2579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "0509e3f4474a4c33b32057309e272b56"
format_stamp: "Formatted at 2026-09-25 21:55:58 on dist-test-slave-2g0f"
server_key: "5b9f2e33c9998686ff6aa0955fa43095"
server_key_iv: "4ee8a1a030f4d41f81e1cd0bbcb5011f"
server_key_version: "encryptionkey@0"
I20260925 21:55:58.907006 2579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "0509e3f4474a4c33b32057309e272b56"
format_stamp: "Formatted at 2026-09-25 21:55:58 on dist-test-slave-2g0f"
server_key: "5b9f2e33c9998686ff6aa0955fa43095"
server_key_iv: "4ee8a1a030f4d41f81e1cd0bbcb5011f"
server_key_version: "encryptionkey@0"
I20260925 21:55:58.908406 2579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:55:58.909224 2594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:55:58.909392 2579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:55:58.909494 2579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "0509e3f4474a4c33b32057309e272b56"
format_stamp: "Formatted at 2026-09-25 21:55:58 on dist-test-slave-2g0f"
server_key: "5b9f2e33c9998686ff6aa0955fa43095"
server_key_iv: "4ee8a1a030f4d41f81e1cd0bbcb5011f"
server_key_version: "encryptionkey@0"
I20260925 21:55:58.909560 2579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:55:58.933671 2579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:55:58.935487 2579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:55:58.935601 2579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:55:58.935801 2579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:55:58.936139 2579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:55:58.936174 2579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:55:58.936200 2579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:55:58.936214 2579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:55:58.942066 2579 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:46713
I20260925 21:55:58.942108 2707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:46713 every 8 connection(s)
I20260925 21:55:58.942414 2579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:55:58.944993 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 2579
I20260925 21:55:58.945083 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:55:58.945236 2331 external_mini_cluster.cc:1428] Setting key 71b50419e3b3acacd5408abf758e1abf
Sep 25 21:55:58 dist-test-slave-2g0f krb5kdc[2338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373358, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:55:58.952275 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:55:58.943201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:52481 (local address 127.2.70.254:35847)
0925 21:55:58.943274 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 21:55:58.943277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:55:58.943448 (+ 171us) server_negotiation.cc:407] Connection header received
0925 21:55:58.944028 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:55:58.944030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:55:58.944041 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:55:58.944094 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:55:58.945128 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:55:58.945808 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:55:58.946806 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:55:58.947081 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:55:58.947150 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:55:58.949568 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:55:58.949569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:55:58.949569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:55:58.949577 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:55:58.950872 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:55:58.951294 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:55:58.951295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:55:58.951296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:55:58.951351 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:55:58.951598 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:55:58.951599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:55:58.951599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:55:58.951716 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:55:58.951774 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 21:55:58.952090 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 21:55:58.952182 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 21:55:58.952448 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:55:58.943158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35847 (local address 127.2.70.194:52481)
0925 21:55:58.943327 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 21:55:58.943368 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 21:55:58.943911 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:55:58.944177 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:55:58.944182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:55:58.944748 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:55:58.945015 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:55:58.945022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:55:58.945887 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:55:58.945888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:55:58.946718 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:55:58.946721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:55:58.947170 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:55:58.947171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:55:58.947297 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:55:58.947924 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:55:58.947932 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:55:58.949469 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:55:58.950961 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:55:58.950968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:55:58.950971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:55:58.951222 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:55:58.951407 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:55:58.951408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:55:58.951408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:55:58.951541 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:55:58.951772 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:55:58.951779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:55:58.951957 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 21:55:58.952058 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 21:55:58.952225 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260925 21:55:58.952875 2708 heartbeater.cc:347] Connected to a master server at 127.2.70.254:35847
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
I20260925 21:55:58.952996 2708 heartbeater.cc:464] Registering TS with master...
I20260925 21:55:58.953202 2708 heartbeater.cc:511] Master 127.2.70.254:35847 requested a full tablet report, sending...
I20260925 21:55:58.953611 2383 ts_manager.cc:194] Registered new tserver with Master: 0509e3f4474a4c33b32057309e272b56 (127.2.70.194:46713)
I20260925 21:55:58.954051 2383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:52481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:55:58.970656 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35847
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:55:59.050302 2713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:55:59.050509 2713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:55:59.050531 2713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:55:59.051635 2713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:55:59.051668 2713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:55:59.051693 2713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:55:59.053130 2713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35847
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:55:59.053313 2713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:55:59.053490 2713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:55:59.058277 2721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:55:59.058283 2718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:55:59.058283 2719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:55:59.058457 2713 server_base.cc:1034] running on GCE node
I20260925 21:55:59.058732 2713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:55:59.058944 2713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:55:59.060071 2713 hybrid_clock.cc:648] HybridClock initialized: now 1790373359060059 us; error 29 us; skew 500 ppm
Sep 25 21:55:59 dist-test-slave-2g0f krb5kdc[2338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373359, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:55:59.063120 2713 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:55:59.063549 2713 webserver.cc:460] Webserver started at http://127.2.70.195:33267/ using document root <none> and password file <none>
I20260925 21:55:59.063724 2713 fs_manager.cc:359] Metadata directory not provided
I20260925 21:55:59.063761 2713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:55:59.063886 2713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:55:59.064899 2713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "7ffeb4b6fb174bec9098ce46d93b5044"
format_stamp: "Formatted at 2026-09-25 21:55:59 on dist-test-slave-2g0f"
server_key: "354ff4e28a5551e8a0fa66d964cab77e"
server_key_iv: "783a31bee286531889c91b1da8b451c6"
server_key_version: "encryptionkey@0"
I20260925 21:55:59.065197 2713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "7ffeb4b6fb174bec9098ce46d93b5044"
format_stamp: "Formatted at 2026-09-25 21:55:59 on dist-test-slave-2g0f"
server_key: "354ff4e28a5551e8a0fa66d964cab77e"
server_key_iv: "783a31bee286531889c91b1da8b451c6"
server_key_version: "encryptionkey@0"
I20260925 21:55:59.066325 2713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:55:59.066960 2728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:55:59.067121 2713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:55:59.067195 2713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "7ffeb4b6fb174bec9098ce46d93b5044"
format_stamp: "Formatted at 2026-09-25 21:55:59 on dist-test-slave-2g0f"
server_key: "354ff4e28a5551e8a0fa66d964cab77e"
server_key_iv: "783a31bee286531889c91b1da8b451c6"
server_key_version: "encryptionkey@0"
I20260925 21:55:59.067250 2713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:55:59.095067 2713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:55:59.097188 2713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:55:59.097304 2713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:55:59.097568 2713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:55:59.097868 2713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:55:59.097905 2713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:55:59.097934 2713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:55:59.097949 2713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:55:59.103699 2713 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:42601
I20260925 21:55:59.103910 2841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:42601 every 8 connection(s)
I20260925 21:55:59.104136 2713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:55:59.105446 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 2713
I20260925 21:55:59.105531 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:55:59.105695 2331 external_mini_cluster.cc:1428] Setting key 1f65dec8a07f7bc28ad04cf34ee09d54
Sep 25 21:55:59 dist-test-slave-2g0f krb5kdc[2338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373359, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:55:59.113365 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:55:59.104805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:55041 (local address 127.2.70.254:35847)
0925 21:55:59.104870 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 21:55:59.104872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:55:59.105066 (+ 194us) server_negotiation.cc:407] Connection header received
0925 21:55:59.105562 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:55:59.105564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:55:59.105576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:55:59.105622 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:55:59.106628 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:55:59.107238 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:55:59.108305 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:55:59.108510 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:55:59.108581 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:55:59.110792 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:55:59.110793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:55:59.110794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:55:59.110802 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:55:59.112210 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:55:59.112591 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:55:59.112592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:55:59.112593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:55:59.112642 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:55:59.112862 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:55:59.112862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:55:59.112862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:55:59.112963 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:55:59.113009 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 21:55:59.113224 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 21:55:59.113286 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 21:55:59.113515 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:55:59.104782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35847 (local address 127.2.70.195:55041)
0925 21:55:59.104960 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 21:55:59.104996 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 21:55:59.105463 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:55:59.105712 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:55:59.105717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:55:59.106347 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:55:59.106549 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:55:59.106555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:55:59.107312 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:55:59.107312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:55:59.108207 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:55:59.108209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:55:59.108580 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:55:59.108581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:55:59.108674 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:55:59.109140 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:55:59.109149 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:55:59.110714 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:55:59.112279 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:55:59.112285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:55:59.112287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:55:59.112531 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:55:59.112691 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:55:59.112691 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:55:59.112691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:55:59.112787 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:55:59.113007 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:55:59.113011 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:55:59.113123 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 21:55:59.113203 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 21:55:59.113319 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260925 21:55:59.113950 2842 heartbeater.cc:347] Connected to a master server at 127.2.70.254:35847
I20260925 21:55:59.114032 2842 heartbeater.cc:464] Registering TS with master...
I20260925 21:55:59.114198 2842 heartbeater.cc:511] Master 127.2.70.254:35847 requested a full tablet report, sending...
I20260925 21:55:59.114609 2383 ts_manager.cc:194] Registered new tserver with Master: 7ffeb4b6fb174bec9098ce46d93b5044 (127.2.70.195:42601)
I20260925 21:55:59.115023 2383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:55041
I20260925 21:55:59.117784 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 21:55:59 dist-test-slave-2g0f krb5kdc[2338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373359, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 21:55:59 dist-test-slave-2g0f krb5kdc[2338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373359, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.193@KRBTEST.COM
I20260925 21:55:59.142470 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:55:59.134918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.2.70.193:33825)
0925 21:55:59.135098 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 21:55:59.135104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 21:55:59.135117 (+ 13us) server_negotiation.cc:407] Connection header received
0925 21:55:59.135149 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:55:59.135151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:55:59.135167 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:55:59.135231 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:55:59.135797 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:55:59.136662 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:55:59.137379 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:55:59.137700 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:55:59.137751 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:55:59.139884 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:55:59.139893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:55:59.139893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:55:59.139901 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:55:59.141276 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:55:59.141587 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:55:59.141589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:55:59.141590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:55:59.141640 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:55:59.141804 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:55:59.141805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:55:59.141805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:55:59.141982 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:55:59.142118 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 21:55:59.142284 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 21:55:59.142360 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
W20260925 21:55:59.142850 2488 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47854
Sep 25 21:55:59 dist-test-slave-2g0f krb5kdc[2338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 21:55:59 dist-test-slave-2g0f krb5kdc[2338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.193@KRBTEST.COM
I20260925 21:55:59.169263 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:55:59.158125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.2.70.193:33825)
0925 21:55:59.158176 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 21:55:59.158178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:55:59.158188 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:55:59.158255 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:55:59.158256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:55:59.158267 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:55:59.158325 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:55:59.158890 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:55:59.159502 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:55:59.160263 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:55:59.160491 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:55:59.160548 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:55:59.167176 (+ 6628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:55:59.167178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:55:59.167178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:55:59.167186 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:55:59.168420 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:55:59.168711 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:55:59.168712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:55:59.168712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:55:59.168762 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:55:59.168897 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:55:59.168897 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:55:59.168898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:55:59.168988 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:55:59.169068 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 21:55:59.169135 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 21:55:59.169196 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 21:55:59.169783 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 2445
I20260925 21:55:59.174009 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 2579
I20260925 21:55:59.178505 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 2713
I20260925 21:55:59.182312 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 2351
2026-09-25T21:55:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1778 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:55:59 dist-test-slave-2g0f krb5kdc[2856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:55:59 dist-test-slave-2g0f krb5kdc[2856](info): set up 2 sockets
Sep 25 21:55:59 dist-test-slave-2g0f krb5kdc[2856](info): commencing operation
krb5kdc: starting...
W20260925 21:56:00.135139 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:00 dist-test-slave-2g0f krb5kdc[2856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:56:00.250200 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:46575
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:46575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:00.329943 2869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:00.330163 2869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:00.330194 2869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:00.331388 2869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:00.331430 2869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:00.331449 2869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:00.331465 2869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:00.331480 2869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:00.333048 2869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:46575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:46575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:00.333303 2869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:00.333521 2869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:00.338176 2877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:00.338183 2875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:00.338359 2874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:00.338387 2869 server_base.cc:1034] running on GCE node
I20260925 21:56:00.338611 2869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:00.338889 2869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:00.340013 2869 hybrid_clock.cc:648] HybridClock initialized: now 1790373360339992 us; error 27 us; skew 500 ppm
Sep 25 21:56:00 dist-test-slave-2g0f krb5kdc[2856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:00.342463 2869 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:56:00.342864 2869 webserver.cc:460] Webserver started at http://127.2.70.254:41845/ using document root <none> and password file <none>
I20260925 21:56:00.343041 2869 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:00.343079 2869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:00.343168 2869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:00.344220 2869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "c6b1faa58c614ce58053a5dda58d2340"
format_stamp: "Formatted at 2026-09-25 21:56:00 on dist-test-slave-2g0f"
server_key: "5966cc1eee6b718d5936baaa9ee73cc7"
server_key_iv: "031bafc2814eba18e21c5c00e68d7d8b"
server_key_version: "encryptionkey@0"
I20260925 21:56:00.344539 2869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "c6b1faa58c614ce58053a5dda58d2340"
format_stamp: "Formatted at 2026-09-25 21:56:00 on dist-test-slave-2g0f"
server_key: "5966cc1eee6b718d5936baaa9ee73cc7"
server_key_iv: "031bafc2814eba18e21c5c00e68d7d8b"
server_key_version: "encryptionkey@0"
I20260925 21:56:00.345737 2869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:00.346405 2884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:00.346551 2869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:00.346637 2869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "c6b1faa58c614ce58053a5dda58d2340"
format_stamp: "Formatted at 2026-09-25 21:56:00 on dist-test-slave-2g0f"
server_key: "5966cc1eee6b718d5936baaa9ee73cc7"
server_key_iv: "031bafc2814eba18e21c5c00e68d7d8b"
server_key_version: "encryptionkey@0"
I20260925 21:56:00.346697 2869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:00.364444 2869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:00.366667 2869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:00.366832 2869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:00.370641 2869 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:46575
I20260925 21:56:00.370677 2936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:46575 every 8 connection(s)
I20260925 21:56:00.371024 2869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:00.372018 2937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:00.374469 2937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340: Bootstrap starting.
I20260925 21:56:00.375012 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 2869
I20260925 21:56:00.375075 2937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:00.375089 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:00.375233 2331 external_mini_cluster.cc:1428] Setting key 734ce634c4415ba7731c9080b4cd16ed
I20260925 21:56:00.375378 2937 log.cc:824] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:00.376283 2937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340: No bootstrap required, opened a new log
I20260925 21:56:00.378774 2937 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b1faa58c614ce58053a5dda58d2340" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46575 } }
I20260925 21:56:00.378953 2937 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:00.378985 2937 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b1faa58c614ce58053a5dda58d2340, State: Initialized, Role: FOLLOWER
I20260925 21:56:00.379127 2937 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b1faa58c614ce58053a5dda58d2340" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46575 } }
I20260925 21:56:00.379207 2937 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:00.379230 2937 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:00.379276 2937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:00.379897 2937 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b1faa58c614ce58053a5dda58d2340" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46575 } }
I20260925 21:56:00.380002 2937 leader_election.cc:302] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6b1faa58c614ce58053a5dda58d2340; no voters:
I20260925 21:56:00.380170 2937 leader_election.cc:288] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:00.380213 2942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:00.380348 2942 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [term 1 LEADER]: Becoming Leader. State: Replica: c6b1faa58c614ce58053a5dda58d2340, State: Running, Role: LEADER
I20260925 21:56:00.380487 2937 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:00.380470 2942 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b1faa58c614ce58053a5dda58d2340" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46575 } }
I20260925 21:56:00.381218 2944 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b1faa58c614ce58053a5dda58d2340. Latest consensus state: current_term: 1 leader_uuid: "c6b1faa58c614ce58053a5dda58d2340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b1faa58c614ce58053a5dda58d2340" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46575 } } }
I20260925 21:56:00.381304 2944 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:00.381508 2951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:00.382064 2951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:00.382756 2943 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6b1faa58c614ce58053a5dda58d2340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b1faa58c614ce58053a5dda58d2340" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46575 } } }
I20260925 21:56:00.382838 2943 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340 [sys.catalog]: This master's current role is: LEADER
Sep 25 21:56:00 dist-test-slave-2g0f krb5kdc[2856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:00.383713 2951 catalog_manager.cc:1347] Generated new cluster ID: 0e4c59b2462e4157a5df9b835a19b6c1
I20260925 21:56:00.383765 2951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:00.386790 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.375899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.2.70.254:46575)
0925 21:56:00.376099 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:00.376108 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:00.376145 (+ 37us) server_negotiation.cc:407] Connection header received
0925 21:56:00.376725 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:00.376736 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:00.376807 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:00.376972 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:00.377567 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.378975 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.379820 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.380505 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.380620 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.383665 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:00.383669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:00.383669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:00.383682 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:00.385339 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:00.385680 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:00.385684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:00.385684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:00.385769 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:00.385939 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:00.385940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:00.385941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:00.386190 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:00.386343 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:00.386437 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 21:56:00.386569 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260925 21:56:00.395666 2951 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:00.396236 2951 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:00.399729 2951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6b1faa58c614ce58053a5dda58d2340: Generated new TSK 0
I20260925 21:56:00.400043 2951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:56:00.432404 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46575
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:00.522173 2963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:00.522382 2963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:00.522405 2963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:00.523599 2963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:00.523643 2963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:00.523670 2963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:00.525256 2963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46575
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:00.525496 2963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:00.525712 2963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:00.531113 2968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:00.531167 2971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:00.531239 2969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:00.531276 2963 server_base.cc:1034] running on GCE node
I20260925 21:56:00.531582 2963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:00.531807 2963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:00.532972 2963 hybrid_clock.cc:648] HybridClock initialized: now 1790373360532960 us; error 25 us; skew 500 ppm
Sep 25 21:56:00 dist-test-slave-2g0f krb5kdc[2856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:00.535571 2963 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:56:00.536053 2963 webserver.cc:460] Webserver started at http://127.2.70.193:46529/ using document root <none> and password file <none>
I20260925 21:56:00.536265 2963 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:00.536309 2963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:00.536437 2963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:00.537524 2963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "c210e5435635485dab7152ba152ae594"
format_stamp: "Formatted at 2026-09-25 21:56:00 on dist-test-slave-2g0f"
server_key: "9077faad2c3ab51a819e0a2109fc2077"
server_key_iv: "71e692c83e6a92ce62871554dbadde29"
server_key_version: "encryptionkey@0"
I20260925 21:56:00.537876 2963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "c210e5435635485dab7152ba152ae594"
format_stamp: "Formatted at 2026-09-25 21:56:00 on dist-test-slave-2g0f"
server_key: "9077faad2c3ab51a819e0a2109fc2077"
server_key_iv: "71e692c83e6a92ce62871554dbadde29"
server_key_version: "encryptionkey@0"
I20260925 21:56:00.539139 2963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:00.540000 2978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:00.540151 2963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:00.540236 2963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "c210e5435635485dab7152ba152ae594"
format_stamp: "Formatted at 2026-09-25 21:56:00 on dist-test-slave-2g0f"
server_key: "9077faad2c3ab51a819e0a2109fc2077"
server_key_iv: "71e692c83e6a92ce62871554dbadde29"
server_key_version: "encryptionkey@0"
I20260925 21:56:00.540302 2963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:00.551780 2963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:00.553956 2963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:00.554109 2963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:00.554329 2963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:00.554661 2963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:00.554701 2963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:00.554773 2963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:00.554803 2963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:00.560308 2963 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:45283
I20260925 21:56:00.560351 3091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:45283 every 8 connection(s)
I20260925 21:56:00.560678 2963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
Sep 25 21:56:00 dist-test-slave-2g0f krb5kdc[2856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:00.568185 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 2963
I20260925 21:56:00.568284 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:00.568445 2331 external_mini_cluster.cc:1428] Setting key ba5dd08706109f30abb4200b23d60a5d
I20260925 21:56:00.570218 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.561443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:47385 (local address 127.2.70.254:46575)
0925 21:56:00.561505 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:00.561507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:00.561658 (+ 151us) server_negotiation.cc:407] Connection header received
0925 21:56:00.562616 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:00.562618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:00.562630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:00.562686 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:00.563286 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.563812 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.564913 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.565121 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.565185 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.567304 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:00.567306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:00.567306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:00.567313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:00.568853 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:00.569278 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:00.569279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:00.569280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:00.569352 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:00.569581 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:00.569581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:00.569581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:00.569700 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:00.569757 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:00.570037 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 21:56:00.570127 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 21:56:00.570461 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.561391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46575 (local address 127.2.70.193:47385)
0925 21:56:00.561543 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:00.561580 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:00.562510 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:00.562747 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:00.562751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:00.563017 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:00.563213 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.563217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.563943 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.563943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.564798 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.564802 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.565196 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.565196 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.565310 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.565832 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:00.565842 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:00.567192 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:00.568923 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:00.568932 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:00.568934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:00.569198 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:00.569411 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:00.569412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:00.569412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:00.569527 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:00.569754 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:00.569762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:00.569918 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 21:56:00.570011 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 21:56:00.570176 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260925 21:56:00.570930 3092 heartbeater.cc:347] Connected to a master server at 127.2.70.254:46575
I20260925 21:56:00.571050 3092 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:00.571252 3092 heartbeater.cc:511] Master 127.2.70.254:46575 requested a full tablet report, sending...
I20260925 21:56:00.571805 2901 ts_manager.cc:194] Registered new tserver with Master: c210e5435635485dab7152ba152ae594 (127.2.70.193:45283)
I20260925 21:56:00.572515 2901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:47385
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:56:00.595417 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46575
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:00.675563 3097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:00.675772 3097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:00.675796 3097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:00.677033 3097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:00.677085 3097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:00.677112 3097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:00.678680 3097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46575
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:00.678917 3097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:00.679145 3097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:00.684230 3105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:00.684247 3097 server_base.cc:1034] running on GCE node
W20260925 21:56:00.684262 3103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:00.684230 3102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:00.684612 3097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:00.684839 3097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:00.685990 3097 hybrid_clock.cc:648] HybridClock initialized: now 1790373360685974 us; error 33 us; skew 500 ppm
Sep 25 21:56:00 dist-test-slave-2g0f krb5kdc[2856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:00.688632 3097 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:56:00.689147 3097 webserver.cc:460] Webserver started at http://127.2.70.194:41923/ using document root <none> and password file <none>
I20260925 21:56:00.689360 3097 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:00.689410 3097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:00.689543 3097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:00.690572 3097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "9109cb40fd9c4e89a8060c19452cba61"
format_stamp: "Formatted at 2026-09-25 21:56:00 on dist-test-slave-2g0f"
server_key: "e688e70c81747e14d9692ab7d12ba169"
server_key_iv: "adb59b47aa46d1f4450d7dce86297e0f"
server_key_version: "encryptionkey@0"
I20260925 21:56:00.690902 3097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "9109cb40fd9c4e89a8060c19452cba61"
format_stamp: "Formatted at 2026-09-25 21:56:00 on dist-test-slave-2g0f"
server_key: "e688e70c81747e14d9692ab7d12ba169"
server_key_iv: "adb59b47aa46d1f4450d7dce86297e0f"
server_key_version: "encryptionkey@0"
I20260925 21:56:00.692227 3097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 21:56:00.692988 3112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:00.693135 3097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 21:56:00.693216 3097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "9109cb40fd9c4e89a8060c19452cba61"
format_stamp: "Formatted at 2026-09-25 21:56:00 on dist-test-slave-2g0f"
server_key: "e688e70c81747e14d9692ab7d12ba169"
server_key_iv: "adb59b47aa46d1f4450d7dce86297e0f"
server_key_version: "encryptionkey@0"
I20260925 21:56:00.693274 3097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:00.700173 3097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:00.702108 3097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:00.702242 3097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:00.702440 3097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:00.702741 3097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:00.702786 3097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:00.702816 3097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:00.702840 3097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:00.708424 3097 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:45555
I20260925 21:56:00.708472 3225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:45555 every 8 connection(s)
I20260925 21:56:00.708781 3097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:00.710131 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 3097
I20260925 21:56:00.710218 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:00.710366 2331 external_mini_cluster.cc:1428] Setting key cca2cd26ab5e543ef343009dfb018b43
Sep 25 21:56:00 dist-test-slave-2g0f krb5kdc[2856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
I20260925 21:56:00.718765 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.709552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:53833 (local address 127.2.70.254:46575)
0925 21:56:00.709634 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:00.709636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:00.709784 (+ 148us) server_negotiation.cc:407] Connection header received
0925 21:56:00.710816 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:00.710817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:00.710829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:00.710884 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:00.711538 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.712121 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.713127 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.713326 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.713383 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.715899 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:00.715900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:00.715900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:00.715908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:00.717271 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:00.717742 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:00.717743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:00.717743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:00.717822 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:00.718088 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:00.718089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:00.718089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:00.718196 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:00.718242 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:00.718621 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 21:56:00.718695 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 21:56:00.718897 3228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.709480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46575 (local address 127.2.70.194:53833)
0925 21:56:00.709659 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:00.709706 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:00.710724 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:00.710954 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:00.710959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:00.711220 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:00.711436 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.711441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.712211 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.712212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.713052 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.713054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.713417 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.713418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.713569 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.714084 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:00.714094 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:00.715733 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:00.717361 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:00.717370 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:00.717373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:00.717666 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:00.717887 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:00.717887 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:00.717888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:00.718029 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:00.718283 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:00.718290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:00.718470 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 21:56:00.718582 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 21:56:00.718721 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260925 21:56:00.719426 3226 heartbeater.cc:347] Connected to a master server at 127.2.70.254:46575
I20260925 21:56:00.719522 3226 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:00.719702 3226 heartbeater.cc:511] Master 127.2.70.254:46575 requested a full tablet report, sending...
I20260925 21:56:00.720117 2901 ts_manager.cc:194] Registered new tserver with Master: 9109cb40fd9c4e89a8060c19452cba61 (127.2.70.194:45555)
I20260925 21:56:00.720506 2901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:53833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:56:00.736614 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46575
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:00.816754 3231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:00.816982 3231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:00.817014 3231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:00.818194 3231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:00.818240 3231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:00.818275 3231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:00.819909 3231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46575
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:00.820155 3231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:00.820370 3231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:00.825129 3239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:00.825137 3237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:00.825363 3231 server_base.cc:1034] running on GCE node
W20260925 21:56:00.825176 3236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:00.825644 3231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:00.825855 3231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:00.826989 3231 hybrid_clock.cc:648] HybridClock initialized: now 1790373360826982 us; error 38 us; skew 500 ppm
Sep 25 21:56:00 dist-test-slave-2g0f krb5kdc[2856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:00.829473 3231 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:56:00.829869 3231 webserver.cc:460] Webserver started at http://127.2.70.195:37709/ using document root <none> and password file <none>
I20260925 21:56:00.830046 3231 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:00.830085 3231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:00.830176 3231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:00.831269 3231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "78b6404d27c04411b4fba291d55a442e"
format_stamp: "Formatted at 2026-09-25 21:56:00 on dist-test-slave-2g0f"
server_key: "d142a2a3d776e417c608861206020934"
server_key_iv: "22328d8a18d10a59486c5fda8e5163fd"
server_key_version: "encryptionkey@0"
I20260925 21:56:00.831583 3231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "78b6404d27c04411b4fba291d55a442e"
format_stamp: "Formatted at 2026-09-25 21:56:00 on dist-test-slave-2g0f"
server_key: "d142a2a3d776e417c608861206020934"
server_key_iv: "22328d8a18d10a59486c5fda8e5163fd"
server_key_version: "encryptionkey@0"
I20260925 21:56:00.832876 3231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:00.833542 3246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:00.833693 3231 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:00.833774 3231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "78b6404d27c04411b4fba291d55a442e"
format_stamp: "Formatted at 2026-09-25 21:56:00 on dist-test-slave-2g0f"
server_key: "d142a2a3d776e417c608861206020934"
server_key_iv: "22328d8a18d10a59486c5fda8e5163fd"
server_key_version: "encryptionkey@0"
I20260925 21:56:00.833864 3231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:00.850198 3231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:00.852181 3231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:00.852294 3231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:00.852488 3231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:00.852751 3231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:00.852779 3231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:00.852800 3231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:00.852826 3231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:00.858136 3231 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:39649
I20260925 21:56:00.858208 3359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:39649 every 8 connection(s)
I20260925 21:56:00.858489 3231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:00.861122 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 3231
I20260925 21:56:00.861220 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:00.861370 2331 external_mini_cluster.cc:1428] Setting key fb688889fd5cce3dec22ac382c28231e
Sep 25 21:56:00 dist-test-slave-2g0f krb5kdc[2856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:00.868323 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.859184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:45779 (local address 127.2.70.254:46575)
0925 21:56:00.859262 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:00.859264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:00.859420 (+ 156us) server_negotiation.cc:407] Connection header received
0925 21:56:00.860477 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:00.860478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:00.860487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:00.860533 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:00.861192 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.861866 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.862913 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.863127 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.863191 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.865407 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:00.865409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:00.865409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:00.865418 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:00.866932 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:00.867342 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:00.867344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:00.867344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:00.867410 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:00.867637 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:00.867638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:00.867638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:00.867775 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:00.867860 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:00.868134 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 21:56:00.868221 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 21:56:00.868539 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.859159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46575 (local address 127.2.70.195:45779)
0925 21:56:00.859301 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:00.859338 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:00.860389 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:00.860600 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:00.860603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:00.860876 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:00.861092 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.861097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.861950 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.861951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.862823 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.862826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.863202 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.863204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.863345 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.863916 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:00.863924 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:00.865309 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:00.867015 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:00.867022 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:00.867024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:00.867270 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:00.867468 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:00.867469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:00.867469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:00.867577 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:00.867856 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:00.867862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:00.868007 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 21:56:00.868106 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 21:56:00.868308 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260925 21:56:00.869153 3360 heartbeater.cc:347] Connected to a master server at 127.2.70.254:46575
I20260925 21:56:00.869246 3360 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:00.869426 3360 heartbeater.cc:511] Master 127.2.70.254:46575 requested a full tablet report, sending...
I20260925 21:56:00.869779 2901 ts_manager.cc:194] Registered new tserver with Master: 78b6404d27c04411b4fba291d55a442e (127.2.70.195:39649)
I20260925 21:56:00.870222 2901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:45779
I20260925 21:56:00.873111 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:56:00.880821 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.874764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56202 (local address 127.2.70.254:46575)
0925 21:56:00.874827 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:00.874829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:00.874955 (+ 126us) server_negotiation.cc:407] Connection header received
0925 21:56:00.874999 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:00.875000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:00.875010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:00.875062 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:00.875584 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.876152 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.876884 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.877074 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.877122 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.878353 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:00.878355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:00.878355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:00.878361 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:00.879687 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:00.880095 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:00.880096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:00.880096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:00.880175 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:00.880333 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:00.880334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:00.880334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:00.880486 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:00.880556 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:00.880686 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 21:56:00.880751 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 21:56:00.883251 2901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:00.884904 2901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:00.890897 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.887164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.2.70.194:45555)
0925 21:56:00.887338 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:00.887343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:00.887354 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:56:00.887415 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:00.887417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:00.887432 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:00.887497 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:00.887627 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.888774 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.889961 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.890516 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.890585 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.890643 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:00.890778 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 21:56:00.890814 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260925 21:56:00.890894 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.887060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:45555 (local address 127.0.0.1:41936)
0925 21:56:00.887215 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:00.887255 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:00.887332 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:00.887498 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:00.887500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:00.887502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:00.887575 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.887577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.888884 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.888885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.889866 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.889869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.890595 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.890596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.890671 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.890686 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 21:56:00.890759 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 21:56:00.890811 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260925 21:56:00.891350 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.887062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:39649 (local address 127.0.0.1:54812)
0925 21:56:00.887324 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:00.887335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:00.887385 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:00.887741 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:00.887742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:00.887744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:00.887792 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.887794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.888996 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.888997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.890070 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.890072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.891152 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.891153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.891202 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.891205 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:00.891247 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 21:56:00.891283 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":55,"threads_started":1}
I20260925 21:56:00.891503 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.887423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.2.70.193:45283)
0925 21:56:00.887998 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:00.888002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:00.888011 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:00.888108 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:00.888110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:00.888126 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:00.888248 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:00.888424 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.889279 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.890517 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.891182 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.891251 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.891305 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:00.891386 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 21:56:00.891421 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":80,"threads_started":1}
I20260925 21:56:00.891705 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.887167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.2.70.195:39649)
0925 21:56:00.887609 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:00.887613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:00.887620 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:56:00.887650 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:00.887651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:00.887663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:00.887729 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:00.887890 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.888899 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.890161 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.890876 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.890931 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.891122 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:00.891588 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 21:56:00.891630 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":76,"threads_started":1}
I20260925 21:56:00.891456 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.887237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:45283 (local address 127.0.0.1:45342)
0925 21:56:00.887465 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:00.887473 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:00.887507 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:00.888264 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:00.888265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:00.888267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:00.888315 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.888316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.889397 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.889398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.890437 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.890439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.891256 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.891257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.891314 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.891317 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:00.891365 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 21:56:00.891396 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":28,"threads_started":1}
I20260925 21:56:00.891778 3160 tablet_service.cc:1467] Processing CreateTablet for tablet 38fa463c132d40f78e089ee63ae51dcb (DEFAULT_TABLE table=test-table [id=0b13afd99ad74f0ba8d056f15e09e0a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:00.892197 3160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38fa463c132d40f78e089ee63ae51dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:00.892514 3294 tablet_service.cc:1467] Processing CreateTablet for tablet 38fa463c132d40f78e089ee63ae51dcb (DEFAULT_TABLE table=test-table [id=0b13afd99ad74f0ba8d056f15e09e0a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:00.892534 3026 tablet_service.cc:1467] Processing CreateTablet for tablet 38fa463c132d40f78e089ee63ae51dcb (DEFAULT_TABLE table=test-table [id=0b13afd99ad74f0ba8d056f15e09e0a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:00.892814 3026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38fa463c132d40f78e089ee63ae51dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:00.892825 3294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38fa463c132d40f78e089ee63ae51dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:00.894586 3374 tablet_bootstrap.cc:492] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61: Bootstrap starting.
I20260925 21:56:00.895154 3374 tablet_bootstrap.cc:654] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:00.895558 3374 log.cc:824] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:00.896083 3377 tablet_bootstrap.cc:492] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e: Bootstrap starting.
I20260925 21:56:00.896292 3374 tablet_bootstrap.cc:492] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61: No bootstrap required, opened a new log
I20260925 21:56:00.896374 3374 ts_tablet_manager.cc:1397] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:00.896746 3377 tablet_bootstrap.cc:654] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:00.897362 3377 log.cc:824] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:00.897888 3376 tablet_bootstrap.cc:492] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594: Bootstrap starting.
I20260925 21:56:00.898193 3377 tablet_bootstrap.cc:492] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e: No bootstrap required, opened a new log
I20260925 21:56:00.898252 3377 ts_tablet_manager.cc:1397] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:56:00.898495 3376 tablet_bootstrap.cc:654] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:00.899014 3376 log.cc:824] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:00.899006 3374 raft_consensus.cc:348] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b6404d27c04411b4fba291d55a442e" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39649 } } peers { permanent_uuid: "c210e5435635485dab7152ba152ae594" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45283 } } peers { permanent_uuid: "9109cb40fd9c4e89a8060c19452cba61" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45555 } }
I20260925 21:56:00.899186 3374 raft_consensus.cc:374] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:00.899228 3374 raft_consensus.cc:729] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9109cb40fd9c4e89a8060c19452cba61, State: Initialized, Role: FOLLOWER
I20260925 21:56:00.899364 3374 consensus_queue.cc:260] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b6404d27c04411b4fba291d55a442e" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39649 } } peers { permanent_uuid: "c210e5435635485dab7152ba152ae594" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45283 } } peers { permanent_uuid: "9109cb40fd9c4e89a8060c19452cba61" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45555 } }
I20260925 21:56:00.899632 3374 ts_tablet_manager.cc:1428] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 21:56:00.899658 3226 heartbeater.cc:503] Master 127.2.70.254:46575 was elected leader, sending a full tablet report...
I20260925 21:56:00.899726 3376 tablet_bootstrap.cc:492] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594: No bootstrap required, opened a new log
I20260925 21:56:00.899782 3376 ts_tablet_manager.cc:1397] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:56:00.900074 3377 raft_consensus.cc:348] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b6404d27c04411b4fba291d55a442e" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39649 } } peers { permanent_uuid: "c210e5435635485dab7152ba152ae594" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45283 } } peers { permanent_uuid: "9109cb40fd9c4e89a8060c19452cba61" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45555 } }
I20260925 21:56:00.900244 3377 raft_consensus.cc:374] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:00.900278 3377 raft_consensus.cc:729] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b6404d27c04411b4fba291d55a442e, State: Initialized, Role: FOLLOWER
I20260925 21:56:00.900390 3377 consensus_queue.cc:260] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b6404d27c04411b4fba291d55a442e" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39649 } } peers { permanent_uuid: "c210e5435635485dab7152ba152ae594" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45283 } } peers { permanent_uuid: "9109cb40fd9c4e89a8060c19452cba61" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45555 } }
I20260925 21:56:00.900641 3377 ts_tablet_manager.cc:1428] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 21:56:00.900761 3360 heartbeater.cc:503] Master 127.2.70.254:46575 was elected leader, sending a full tablet report...
I20260925 21:56:00.901652 3376 raft_consensus.cc:348] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b6404d27c04411b4fba291d55a442e" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39649 } } peers { permanent_uuid: "c210e5435635485dab7152ba152ae594" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45283 } } peers { permanent_uuid: "9109cb40fd9c4e89a8060c19452cba61" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45555 } }
I20260925 21:56:00.901788 3376 raft_consensus.cc:374] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:00.901808 3376 raft_consensus.cc:729] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c210e5435635485dab7152ba152ae594, State: Initialized, Role: FOLLOWER
I20260925 21:56:00.901922 3376 consensus_queue.cc:260] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b6404d27c04411b4fba291d55a442e" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39649 } } peers { permanent_uuid: "c210e5435635485dab7152ba152ae594" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45283 } } peers { permanent_uuid: "9109cb40fd9c4e89a8060c19452cba61" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45555 } }
I20260925 21:56:00.902148 3376 ts_tablet_manager.cc:1428] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 21:56:00.902175 3092 heartbeater.cc:503] Master 127.2.70.254:46575 was elected leader, sending a full tablet report...
W20260925 21:56:00.959408 3227 tablet.cc:2369] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:00.965751 3380 raft_consensus.cc:482] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:00.965950 3380 raft_consensus.cc:504] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b6404d27c04411b4fba291d55a442e" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39649 } } peers { permanent_uuid: "c210e5435635485dab7152ba152ae594" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45283 } } peers { permanent_uuid: "9109cb40fd9c4e89a8060c19452cba61" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45555 } }
I20260925 21:56:00.966332 3380 leader_election.cc:288] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b6404d27c04411b4fba291d55a442e (127.2.70.195:39649), c210e5435635485dab7152ba152ae594 (127.2.70.193:45283)
I20260925 21:56:00.970239 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.966549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:60185 (local address 127.2.70.195:39649)
0925 21:56:00.966601 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:00.966603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:00.966615 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:56:00.966736 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:00.966738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:00.966750 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:00.966832 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:00.966980 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.967734 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.969281 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.969878 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.969957 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.969965 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:00.970127 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 21:56:00.970166 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 21:56:00.970239 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.966683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:37043 (local address 127.2.70.193:45283)
0925 21:56:00.966733 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:00.966734 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:00.966777 (+ 43us) server_negotiation.cc:407] Connection header received
0925 21:56:00.966922 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:00.966922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:00.966931 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:00.966984 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:00.967086 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.967761 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.969281 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.969888 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.969957 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.969965 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:00.970127 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 21:56:00.970166 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 21:56:00.970265 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.966579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:45283 (local address 127.2.70.194:37043)
0925 21:56:00.966734 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:00.966743 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:00.966781 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:00.966994 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:00.966995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:00.966997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:00.967044 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.967045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.967898 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.967898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.969169 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.969172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.969964 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.969965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.970052 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.970057 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:56:00.970114 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 21:56:00.970159 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":33,"thread_start_us":116,"threads_started":1}
I20260925 21:56:00.970271 3228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.966480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:39649 (local address 127.2.70.194:60185)
0925 21:56:00.966556 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:00.966568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:00.966610 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:00.966837 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:00.966838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:00.966840 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:00.966912 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.966914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.967862 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.967863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.969169 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:00.969172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.969964 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.969965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:00.970063 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.970066 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:00.970114 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 21:56:00.970160 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":71}
I20260925 21:56:00.970675 3046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fa463c132d40f78e089ee63ae51dcb" candidate_uuid: "9109cb40fd9c4e89a8060c19452cba61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c210e5435635485dab7152ba152ae594" is_pre_election: true
I20260925 21:56:00.970839 3046 raft_consensus.cc:2454] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9109cb40fd9c4e89a8060c19452cba61 in term 0.
I20260925 21:56:00.970799 3314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fa463c132d40f78e089ee63ae51dcb" candidate_uuid: "9109cb40fd9c4e89a8060c19452cba61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b6404d27c04411b4fba291d55a442e" is_pre_election: true
I20260925 21:56:00.971015 3314 raft_consensus.cc:2454] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9109cb40fd9c4e89a8060c19452cba61 in term 0.
I20260925 21:56:00.971069 3116 leader_election.cc:302] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9109cb40fd9c4e89a8060c19452cba61, c210e5435635485dab7152ba152ae594; no voters:
I20260925 21:56:00.971184 3380 raft_consensus.cc:2781] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:00.971235 3380 raft_consensus.cc:482] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:00.971272 3380 raft_consensus.cc:3037] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:00.972005 3380 raft_consensus.cc:504] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b6404d27c04411b4fba291d55a442e" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39649 } } peers { permanent_uuid: "c210e5435635485dab7152ba152ae594" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45283 } } peers { permanent_uuid: "9109cb40fd9c4e89a8060c19452cba61" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45555 } }
I20260925 21:56:00.972119 3380 leader_election.cc:288] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [CANDIDATE]: Term 1 election: Requested vote from peers 78b6404d27c04411b4fba291d55a442e (127.2.70.195:39649), c210e5435635485dab7152ba152ae594 (127.2.70.193:45283)
I20260925 21:56:00.972263 3314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fa463c132d40f78e089ee63ae51dcb" candidate_uuid: "9109cb40fd9c4e89a8060c19452cba61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b6404d27c04411b4fba291d55a442e"
I20260925 21:56:00.972270 3046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fa463c132d40f78e089ee63ae51dcb" candidate_uuid: "9109cb40fd9c4e89a8060c19452cba61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c210e5435635485dab7152ba152ae594"
I20260925 21:56:00.972339 3314 raft_consensus.cc:3037] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:00.972339 3046 raft_consensus.cc:3037] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:00.973181 3314 raft_consensus.cc:2454] T 38fa463c132d40f78e089ee63ae51dcb P 78b6404d27c04411b4fba291d55a442e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9109cb40fd9c4e89a8060c19452cba61 in term 1.
I20260925 21:56:00.973181 3046 raft_consensus.cc:2454] T 38fa463c132d40f78e089ee63ae51dcb P c210e5435635485dab7152ba152ae594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9109cb40fd9c4e89a8060c19452cba61 in term 1.
I20260925 21:56:00.973349 3116 leader_election.cc:302] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9109cb40fd9c4e89a8060c19452cba61, c210e5435635485dab7152ba152ae594; no voters:
I20260925 21:56:00.973495 3380 raft_consensus.cc:2781] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:00.973675 3380 raft_consensus.cc:686] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [term 1 LEADER]: Becoming Leader. State: Replica: 9109cb40fd9c4e89a8060c19452cba61, State: Running, Role: LEADER
I20260925 21:56:00.973770 3380 consensus_queue.cc:237] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b6404d27c04411b4fba291d55a442e" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39649 } } peers { permanent_uuid: "c210e5435635485dab7152ba152ae594" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45283 } } peers { permanent_uuid: "9109cb40fd9c4e89a8060c19452cba61" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45555 } }
I20260925 21:56:00.974642 2901 catalog_manager.cc:5495] T 38fa463c132d40f78e089ee63ae51dcb P 9109cb40fd9c4e89a8060c19452cba61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9109cb40fd9c4e89a8060c19452cba61 (127.2.70.194). New cstate: current_term: 1 leader_uuid: "9109cb40fd9c4e89a8060c19452cba61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b6404d27c04411b4fba291d55a442e" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c210e5435635485dab7152ba152ae594" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9109cb40fd9c4e89a8060c19452cba61" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45555 } health_report { overall_health: HEALTHY } } }
Sep 25 21:56:00 dist-test-slave-2g0f krb5kdc[2856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.193@KRBTEST.COM
I20260925 21:56:00.991211 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:00.983939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.2.70.193:45283)
0925 21:56:00.983989 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:00.983992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:00.984122 (+ 130us) server_negotiation.cc:407] Connection header received
0925 21:56:00.984191 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:00.984192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:00.984202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:00.984245 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:00.984813 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.985289 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.985967 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:00.986159 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:00.986213 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:00.988536 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:00.988552 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:00.988553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:00.988560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:00.990044 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:00.990410 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:00.990414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:00.990414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:00.990468 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:00.990637 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:00.990637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:00.990638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:00.990781 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:00.990979 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:00.991040 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 21:56:00.991119 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 21:56:00 dist-test-slave-2g0f krb5kdc[2856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373360, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 21:56:01 dist-test-slave-2g0f krb5kdc[2856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373360, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.193@KRBTEST.COM
I20260925 21:56:01.013361 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:01.006357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.2.70.193:45283)
0925 21:56:01.006418 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:01.006420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:01.006515 (+ 95us) server_negotiation.cc:407] Connection header received
0925 21:56:01.006560 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:01.006561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:01.006572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:01.006616 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:01.007154 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:01.007636 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:01.008441 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:01.008660 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:01.008723 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:01.010829 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:01.010830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:01.010830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:01.010838 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:01.012265 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:01.012628 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:01.012629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:01.012630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:01.012690 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:01.012837 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:01.012837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:01.012837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:01.012928 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:01.013015 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:01.013169 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 21:56:01.013251 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 21:56:01 dist-test-slave-2g0f krb5kdc[2856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 21:56:01 dist-test-slave-2g0f krb5kdc[2856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.193@KRBTEST.COM
I20260925 21:56:01.034942 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:01.028275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.2.70.193:45283)
0925 21:56:01.028335 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:01.028338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:01.028408 (+ 70us) server_negotiation.cc:407] Connection header received
0925 21:56:01.028459 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:01.028460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:01.028469 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:01.028517 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:01.029085 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:01.029555 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:01.030291 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:01.030480 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:01.030530 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:01.032582 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:01.032583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:01.032584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:01.032591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:01.033895 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:01.034219 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:01.034220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:01.034221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:01.034280 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:01.034429 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:01.034430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:01.034430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:01.034548 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:01.034622 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:01.034716 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 21:56:01.034844 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 21:56:01.035427 3006 tablet_service.cc:2908] Rejecting scan request for tablet 38fa463c132d40f78e089ee63ae51dcb: Uninitialized: clean time has not yet been initialized
I20260925 21:56:01.036242 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 2963
I20260925 21:56:01.040941 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 3097
I20260925 21:56:01.045673 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 3231
I20260925 21:56:01.049345 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 2869
2026-09-25T21:56:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1867 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:01 dist-test-slave-2g0f krb5kdc[3398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:01 dist-test-slave-2g0f krb5kdc[3398](info): set up 2 sockets
Sep 25 21:56:01 dist-test-slave-2g0f krb5kdc[3398](info): commencing operation
krb5kdc: starting...
W20260925 21:56:01.999198 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:02 dist-test-slave-2g0f krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:56:02.115039 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:35175
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:35175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:02.195550 3411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:02.195780 3411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:02.195803 3411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:02.197006 3411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:02.197050 3411 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 21:56:02.197063 3411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:02.197074 3411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:02.197086 3411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:02.197096 3411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:02.198544 3411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:35175
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:35175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:02.198753 3411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:02.198931 3411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:02.204176 3416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:02.204162 3419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:02.204166 3411 server_base.cc:1034] running on GCE node
W20260925 21:56:02.204187 3417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:02.204691 3411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:02.205039 3411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:02.206193 3411 hybrid_clock.cc:648] HybridClock initialized: now 1790373362206175 us; error 44 us; skew 500 ppm
Sep 25 21:56:02 dist-test-slave-2g0f krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:02.209180 3411 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:56:02.209683 3411 webserver.cc:460] Webserver started at http://127.2.70.254:37085/ using document root <none> and password file <none>
I20260925 21:56:02.209894 3411 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:02.209944 3411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:02.210073 3411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:02.211161 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "2333b7aa326f4987b46c1e1f7a400765"
format_stamp: "Formatted at 2026-09-25 21:56:02 on dist-test-slave-2g0f"
server_key: "2616393f13b6bec0488911b3b13d87b6"
server_key_iv: "f7b5c84a09de88a8c77b2d2812f7ca9d"
server_key_version: "encryptionkey@0"
I20260925 21:56:02.211534 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "2333b7aa326f4987b46c1e1f7a400765"
format_stamp: "Formatted at 2026-09-25 21:56:02 on dist-test-slave-2g0f"
server_key: "2616393f13b6bec0488911b3b13d87b6"
server_key_iv: "f7b5c84a09de88a8c77b2d2812f7ca9d"
server_key_version: "encryptionkey@0"
I20260925 21:56:02.213133 3411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:02.213958 3426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:02.214171 3411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:02.214267 3411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "2333b7aa326f4987b46c1e1f7a400765"
format_stamp: "Formatted at 2026-09-25 21:56:02 on dist-test-slave-2g0f"
server_key: "2616393f13b6bec0488911b3b13d87b6"
server_key_iv: "f7b5c84a09de88a8c77b2d2812f7ca9d"
server_key_version: "encryptionkey@0"
I20260925 21:56:02.214340 3411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:02.227068 3411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:02.229537 3411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:02.229691 3411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:02.234025 3411 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:35175
I20260925 21:56:02.234109 3488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:35175 every 8 connection(s)
I20260925 21:56:02.234408 3411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:02.235069 3489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:02.237493 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765: Bootstrap starting.
I20260925 21:56:02.238117 3489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:02.238402 3489 log.cc:824] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:02.239104 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765: No bootstrap required, opened a new log
I20260925 21:56:02.240000 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 3411
I20260925 21:56:02.240072 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:02.240244 2331 external_mini_cluster.cc:1428] Setting key 0c3c1315399c94ea62a33b999b17ad9c
I20260925 21:56:02.241024 3489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2333b7aa326f4987b46c1e1f7a400765" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35175 } }
I20260925 21:56:02.241168 3489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:02.241189 3489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2333b7aa326f4987b46c1e1f7a400765, State: Initialized, Role: FOLLOWER
I20260925 21:56:02.241324 3489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2333b7aa326f4987b46c1e1f7a400765" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35175 } }
I20260925 21:56:02.241391 3489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:02.241412 3489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:02.241456 3489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:02.241968 3489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2333b7aa326f4987b46c1e1f7a400765" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35175 } }
I20260925 21:56:02.242065 3489 leader_election.cc:302] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2333b7aa326f4987b46c1e1f7a400765; no voters:
I20260925 21:56:02.242228 3489 leader_election.cc:288] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:02.242255 3494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:02.242365 3494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [term 1 LEADER]: Becoming Leader. State: Replica: 2333b7aa326f4987b46c1e1f7a400765, State: Running, Role: LEADER
I20260925 21:56:02.242447 3494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2333b7aa326f4987b46c1e1f7a400765" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35175 } }
I20260925 21:56:02.242502 3489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:02.242774 3495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2333b7aa326f4987b46c1e1f7a400765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2333b7aa326f4987b46c1e1f7a400765" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35175 } } }
I20260925 21:56:02.243002 3495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:02.242944 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2333b7aa326f4987b46c1e1f7a400765. Latest consensus state: current_term: 1 leader_uuid: "2333b7aa326f4987b46c1e1f7a400765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2333b7aa326f4987b46c1e1f7a400765" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35175 } } }
I20260925 21:56:02.243229 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765 [sys.catalog]: This master's current role is: LEADER
W20260925 21:56:02.244161 3510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:56:02.244205 3510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:56:02.244256 3503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:02.244395 3503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:02.245873 3503 catalog_manager.cc:1347] Generated new cluster ID: 2735e774b2a148fbb1ab978d43acde32
I20260925 21:56:02.245926 3503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 21:56:02 dist-test-slave-2g0f krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:02.251075 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.240929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.2.70.254:35175)
0925 21:56:02.241097 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.241105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.241130 (+ 25us) server_negotiation.cc:407] Connection header received
0925 21:56:02.241278 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.241284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.241342 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.241496 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:02.242207 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.243581 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.244642 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.245261 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.245375 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.247588 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:02.247592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:02.247593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:02.247606 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:02.249637 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.249977 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.249980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.249980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.250085 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.250260 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.250261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.250261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.250521 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:02.250676 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.250751 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.250909 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260925 21:56:02.258930 3503 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:02.259668 3503 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:02.262439 3503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765: Generated new TSK 0
I20260925 21:56:02.262645 3503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:56:02.298666 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35175
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:02.393200 3515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:02.393422 3515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:02.393446 3515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:02.394658 3515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:02.394779 3515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:02.394801 3515 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 21:56:02.394819 3515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:02.396386 3515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.254:35175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:02.396629 3515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:02.396855 3515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:02.401983 3521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:02.401993 3523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:02.402148 3515 server_base.cc:1034] running on GCE node
W20260925 21:56:02.401983 3520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:02.402488 3515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:02.402731 3515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:02.403901 3515 hybrid_clock.cc:648] HybridClock initialized: now 1790373362403879 us; error 27 us; skew 500 ppm
Sep 25 21:56:02 dist-test-slave-2g0f krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:02.406414 3515 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:56:02.406855 3515 webserver.cc:460] Webserver started at http://127.2.70.193:36633/ using document root <none> and password file <none>
I20260925 21:56:02.407059 3515 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:02.407111 3515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:02.407236 3515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:02.408413 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "121c9199ec25465d9a375a11a0d0568d"
format_stamp: "Formatted at 2026-09-25 21:56:02 on dist-test-slave-2g0f"
server_key: "6dd686c420f78e5d5f4c3be47c4a0d6a"
server_key_iv: "034a5afe7f65b8c689322d5ad93d6cfa"
server_key_version: "encryptionkey@0"
I20260925 21:56:02.408751 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "121c9199ec25465d9a375a11a0d0568d"
format_stamp: "Formatted at 2026-09-25 21:56:02 on dist-test-slave-2g0f"
server_key: "6dd686c420f78e5d5f4c3be47c4a0d6a"
server_key_iv: "034a5afe7f65b8c689322d5ad93d6cfa"
server_key_version: "encryptionkey@0"
I20260925 21:56:02.410100 3515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:02.410964 3530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:02.411158 3515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:02.411244 3515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "121c9199ec25465d9a375a11a0d0568d"
format_stamp: "Formatted at 2026-09-25 21:56:02 on dist-test-slave-2g0f"
server_key: "6dd686c420f78e5d5f4c3be47c4a0d6a"
server_key_iv: "034a5afe7f65b8c689322d5ad93d6cfa"
server_key_version: "encryptionkey@0"
I20260925 21:56:02.411306 3515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:02.446898 3515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:02.448861 3515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:02.448999 3515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:02.449542 3515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:02.449582 3515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:02.449612 3515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:02.449630 3515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 21:56:02 dist-test-slave-2g0f krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:02.455997 3515 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:35563
I20260925 21:56:02.456076 3645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:35563 every 8 connection(s)
I20260925 21:56:02.456367 3515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:02.456671 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 3515
I20260925 21:56:02.456761 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:02.456938 2331 external_mini_cluster.cc:1428] Setting key 47fcacee0adda477756611ce56602740
I20260925 21:56:02.459233 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.450057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:43809 (local address 127.2.70.254:35175)
0925 21:56:02.450113 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.450115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.450341 (+ 226us) server_negotiation.cc:407] Connection header received
0925 21:56:02.450910 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.450911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.450930 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.450992 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:02.451920 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.452693 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.453747 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.453973 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.454029 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.456210 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:02.456212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:02.456212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:02.456220 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:02.457705 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.458230 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.458230 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.458231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.458293 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.458555 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.458556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.458556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.458669 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:02.458718 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.459045 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.459141 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 21:56:02.459398 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.450017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35175 (local address 127.2.70.193:43809)
0925 21:56:02.450194 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:02.450233 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:02.450775 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:02.451093 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:02.451097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:02.451487 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:02.451771 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.451777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.452798 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.452799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.453663 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.453666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.454063 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.454064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.454188 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.454646 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:02.454655 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:02.456103 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:02.457862 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:02.457870 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:02.457872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:02.458149 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:02.458377 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:02.458377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:02.458378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:02.458502 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:02.458721 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:02.458725 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:02.458884 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 21:56:02.459003 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 21:56:02.459174 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260925 21:56:02.459856 3646 heartbeater.cc:347] Connected to a master server at 127.2.70.254:35175
I20260925 21:56:02.459942 3646 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:02.460114 3646 heartbeater.cc:511] Master 127.2.70.254:35175 requested a full tablet report, sending...
I20260925 21:56:02.460631 3442 ts_manager.cc:194] Registered new tserver with Master: 121c9199ec25465d9a375a11a0d0568d (127.2.70.193:35563)
I20260925 21:56:02.461485 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:43809
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
I20260925 21:56:02.466619 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.460900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:52465 (local address 127.2.70.254:35175)
0925 21:56:02.460938 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.460940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.461010 (+ 70us) server_negotiation.cc:407] Connection header received
0925 21:56:02.461067 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.461068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.461077 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.461119 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:02.461417 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.462035 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.463041 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.463221 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.463300 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.464308 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:02.464309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:02.464310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:02.464316 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:02.465562 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.465959 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.465960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.465960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.466017 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.466199 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.466200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.466200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.466308 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:02.466357 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.466493 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.466551 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 21:56:02.466621 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.460844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35175 (local address 127.2.70.193:52465)
0925 21:56:02.460963 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:02.460973 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:02.461024 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:02.461131 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:02.461132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:02.461303 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:02.461354 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.461355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.462138 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.462139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.462956 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.462959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.463303 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.463304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.463390 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.463794 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:02.463799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:02.464214 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:02.465640 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:02.465641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:02.465642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:02.465846 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:02.466079 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:02.466080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:02.466080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:02.466143 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:02.466375 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:02.466376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:02.466416 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 21:56:02.466465 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 21:56:02.466548 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:56:02.483075 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35175
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:02.563313 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:02.563537 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:02.563571 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:02.564816 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:02.564862 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:02.564891 3655 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 21:56:02.564914 3655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:02.566413 3655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.254:35175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:02.566659 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:02.566883 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:56:02.572022 3655 server_base.cc:1034] running on GCE node
W20260925 21:56:02.572002 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:02.571991 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:02.571998 3663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:02.572424 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:02.572670 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:02.573858 3655 hybrid_clock.cc:648] HybridClock initialized: now 1790373362573846 us; error 27 us; skew 500 ppm
Sep 25 21:56:02 dist-test-slave-2g0f krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:02.576488 3655 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:56:02.577003 3655 webserver.cc:460] Webserver started at http://127.2.70.194:42305/ using document root <none> and password file <none>
I20260925 21:56:02.577227 3655 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:02.577287 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:02.577423 3655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:02.578577 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46"
format_stamp: "Formatted at 2026-09-25 21:56:02 on dist-test-slave-2g0f"
server_key: "d46c41e32c729a27dbbca4c7d0006db0"
server_key_iv: "97ff0497041c8f762b803590d8708c2e"
server_key_version: "encryptionkey@0"
I20260925 21:56:02.578922 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46"
format_stamp: "Formatted at 2026-09-25 21:56:02 on dist-test-slave-2g0f"
server_key: "d46c41e32c729a27dbbca4c7d0006db0"
server_key_iv: "97ff0497041c8f762b803590d8708c2e"
server_key_version: "encryptionkey@0"
I20260925 21:56:02.580278 3655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:02.581009 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:02.581168 3655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:02.581254 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46"
format_stamp: "Formatted at 2026-09-25 21:56:02 on dist-test-slave-2g0f"
server_key: "d46c41e32c729a27dbbca4c7d0006db0"
server_key_iv: "97ff0497041c8f762b803590d8708c2e"
server_key_version: "encryptionkey@0"
I20260925 21:56:02.581315 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:02.600507 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:02.602617 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:02.602770 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:02.603264 3655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:02.603299 3655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:02.603336 3655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:02.603361 3655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:02.609558 3655 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:45299
I20260925 21:56:02.609661 3785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:45299 every 8 connection(s)
I20260925 21:56:02.609921 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
Sep 25 21:56:02 dist-test-slave-2g0f krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:02.613083 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.603788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:49305 (local address 127.2.70.254:35175)
0925 21:56:02.603857 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.603859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.604077 (+ 218us) server_negotiation.cc:407] Connection header received
0925 21:56:02.604586 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.604587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.604600 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.604646 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:02.605554 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.606321 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.607780 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.608042 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.608093 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.610614 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:02.610615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:02.610616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:02.610623 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:02.611971 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.612347 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.612348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.612348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.612401 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.612591 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.612592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.612592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.612692 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:02.612738 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.612962 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.613020 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 21:56:02.613260 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.603786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35175 (local address 127.2.70.194:49305)
0925 21:56:02.603963 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:02.603999 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:02.604486 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:02.604787 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:02.604792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:02.605156 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:02.605447 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.605454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.606416 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.606417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.607679 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.607683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.608120 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.608121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.608290 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.608818 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:02.608828 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:02.610526 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:02.612055 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:02.612060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:02.612062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:02.612278 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:02.612451 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:02.612451 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:02.612451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:02.612549 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:02.612741 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:02.612745 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:02.612860 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 21:56:02.612941 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 21:56:02.613054 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260925 21:56:02.613700 3786 heartbeater.cc:347] Connected to a master server at 127.2.70.254:35175
I20260925 21:56:02.613780 3786 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:02.613953 3786 heartbeater.cc:511] Master 127.2.70.254:35175 requested a full tablet report, sending...
I20260925 21:56:02.614342 3443 ts_manager.cc:194] Registered new tserver with Master: 84b026ab7c0d4de9a5c1138c8fbd0e46 (127.2.70.194:45299)
I20260925 21:56:02.614807 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:49305
I20260925 21:56:02.617966 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 3655
I20260925 21:56:02.618064 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:02.618235 2331 external_mini_cluster.cc:1428] Setting key fe466bc90658b00df1968eedfa2a479a
I20260925 21:56:02.620260 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.614604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35175 (local address 127.2.70.194:56541)
0925 21:56:02.614720 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:02.614730 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:02.614793 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:02.614900 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:02.614901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:02.615079 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:02.615197 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.615200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.615818 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.615819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.616352 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.616354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.616680 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.616680 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.616733 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.617157 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:02.617161 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:02.617438 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:02.619289 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:02.619290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:02.619290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:02.619448 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:02.619617 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:02.619618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:02.619618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:02.619673 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:02.620007 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:02.620008 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:02.620053 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 21:56:02.620110 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 21:56:02.620191 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":66,"threads_started":1}
I20260925 21:56:02.620306 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.614674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:56541 (local address 127.2.70.254:35175)
0925 21:56:02.614716 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.614717 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.614779 (+ 62us) server_negotiation.cc:407] Connection header received
0925 21:56:02.614836 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.614837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.614844 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.614897 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:02.615256 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.615739 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.616436 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.616617 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.616674 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.617529 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:02.617530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:02.617530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:02.617537 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:02.619205 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.619511 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.619512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.619513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.619566 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.619722 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.619723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.619723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.619935 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:02.620003 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.620150 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.620221 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:56:02.642299 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35175
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:02.721906 3795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:02.722121 3795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:02.722144 3795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:02.723344 3795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:02.723390 3795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:02.723409 3795 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 21:56:02.723425 3795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:02.724907 3795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.254:35175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:02.725116 3795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:02.725304 3795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:02.730450 3801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:02.730445 3800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:02.730450 3803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:02.730482 3795 server_base.cc:1034] running on GCE node
I20260925 21:56:02.730849 3795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:02.731101 3795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:02.732239 3795 hybrid_clock.cc:648] HybridClock initialized: now 1790373362732222 us; error 32 us; skew 500 ppm
Sep 25 21:56:02 dist-test-slave-2g0f krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:02.734827 3795 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:56:02.735297 3795 webserver.cc:460] Webserver started at http://127.2.70.195:45827/ using document root <none> and password file <none>
I20260925 21:56:02.735500 3795 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:02.735548 3795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:02.735671 3795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:02.736810 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "38af88947c234739b31924fb1eef8a2f"
format_stamp: "Formatted at 2026-09-25 21:56:02 on dist-test-slave-2g0f"
server_key: "65f8d37ae6c6585c3c60fac68b778b30"
server_key_iv: "7839e8653f61781433efe1c3b6eeff0a"
server_key_version: "encryptionkey@0"
I20260925 21:56:02.737134 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "38af88947c234739b31924fb1eef8a2f"
format_stamp: "Formatted at 2026-09-25 21:56:02 on dist-test-slave-2g0f"
server_key: "65f8d37ae6c6585c3c60fac68b778b30"
server_key_iv: "7839e8653f61781433efe1c3b6eeff0a"
server_key_version: "encryptionkey@0"
I20260925 21:56:02.738384 3795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:02.739153 3810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:02.739295 3795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:02.739373 3795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "38af88947c234739b31924fb1eef8a2f"
format_stamp: "Formatted at 2026-09-25 21:56:02 on dist-test-slave-2g0f"
server_key: "65f8d37ae6c6585c3c60fac68b778b30"
server_key_iv: "7839e8653f61781433efe1c3b6eeff0a"
server_key_version: "encryptionkey@0"
I20260925 21:56:02.739435 3795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:02.749115 3795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:02.751215 3795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:02.751344 3795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:02.751901 3795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:02.751943 3795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:02.751964 3795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:02.751976 3795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 21:56:02 dist-test-slave-2g0f krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:02.758613 3795 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:39735
I20260925 21:56:02.758687 3925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:39735 every 8 connection(s)
I20260925 21:56:02.758935 3795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:02.760983 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.752311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:60139 (local address 127.2.70.254:35175)
0925 21:56:02.752373 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.752375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.752545 (+ 170us) server_negotiation.cc:407] Connection header received
0925 21:56:02.753106 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.753108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.753121 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.753189 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:02.753991 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.754541 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.755553 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.755757 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.755813 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.758151 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:02.758152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:02.758153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:02.758161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:02.759620 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.760076 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.760077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.760077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.760141 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.760377 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.760378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.760378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.760495 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:02.760550 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.760817 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.760895 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 21:56:02.761144 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.752273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35175 (local address 127.2.70.195:60139)
0925 21:56:02.752422 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:02.752462 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:02.752996 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:02.753346 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:02.753352 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:02.753606 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:02.753900 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.753905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.754620 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.754622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.755473 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.755475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.755858 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.755859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.755972 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.756470 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:02.756480 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:02.758059 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:02.759701 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:02.759708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:02.759711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:02.759993 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:02.760196 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:02.760197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:02.760197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:02.760307 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:02.760548 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:02.760555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:02.760690 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 21:56:02.760787 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 21:56:02.760939 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260925 21:56:02.761569 3926 heartbeater.cc:347] Connected to a master server at 127.2.70.254:35175
I20260925 21:56:02.761641 3926 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:02.761824 3926 heartbeater.cc:511] Master 127.2.70.254:35175 requested a full tablet report, sending...
I20260925 21:56:02.762179 3442 ts_manager.cc:194] Registered new tserver with Master: 38af88947c234739b31924fb1eef8a2f (127.2.70.195:39735)
I20260925 21:56:02.762725 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:60139
I20260925 21:56:02.767041 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 3795
I20260925 21:56:02.767131 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:02.767287 2331 external_mini_cluster.cc:1428] Setting key 4fd2f950ccec7276164ad0eca15da11a
I20260925 21:56:02.767922 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.762674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:34675 (local address 127.2.70.254:35175)
0925 21:56:02.762712 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.762714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.762776 (+ 62us) server_negotiation.cc:407] Connection header received
0925 21:56:02.762850 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.762851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.762861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.762910 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:02.763188 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.763644 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.764366 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.764547 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.764605 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.765460 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:02.765461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:02.765461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:02.765467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:02.766741 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.767055 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.767056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.767056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.767116 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.767412 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.767413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.767413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.767542 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:02.767596 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.767740 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.767815 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 21:56:02.767958 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.762602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35175 (local address 127.2.70.195:34675)
0925 21:56:02.762710 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:02.762720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:02.762789 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:02.762915 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:02.762916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:02.763057 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:02.763122 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.763124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.763725 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.763725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.764275 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.764277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.764615 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.764616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.764673 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.765095 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:02.765099 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:02.765373 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:02.766824 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:02.766825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:02.766825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:02.766978 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:02.767241 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:02.767242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:02.767243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:02.767318 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:02.767591 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:02.767592 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:02.767642 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 21:56:02.767711 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 21:56:02.767819 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":63,"threads_started":1}
I20260925 21:56:02.768250 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 21:56:02 dist-test-slave-2g0f krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 21:56:02 dist-test-slave-2g0f krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:02.799235 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.791449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.2.70.254:35175)
0925 21:56:02.791506 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.791508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.791519 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:56:02.791550 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.791551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.791562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.791649 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:02.792482 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.793215 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.794294 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.794486 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.794554 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.796826 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:02.796828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:02.796828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:02.796836 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:02.798131 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.798506 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.798508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.798508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.798578 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:02.798761 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:02.798761 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:02.798762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:02.798882 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:02.798956 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.799082 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.799154 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 21:56:02.800884 3442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:02.802397 3442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:02.808755 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.804855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:35563 (local address 127.0.0.1:38196)
0925 21:56:02.805108 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:02.805116 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:02.805157 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:02.805438 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:02.805439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:02.805442 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:02.805498 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.805500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.806880 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.806881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.807860 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.807862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.808507 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.808508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.808557 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.808559 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:02.808598 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 21:56:02.808650 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":84,"threads_started":1}
I20260925 21:56:02.808777 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.804792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.2.70.194:45299)
0925 21:56:02.804995 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.805001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.805027 (+ 26us) server_negotiation.cc:407] Connection header received
0925 21:56:02.805071 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.805074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.805104 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.805182 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:02.805437 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.806344 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.807600 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.808371 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.808438 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.808494 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.808640 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.808677 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
I20260925 21:56:02.809115 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.804991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.2.70.193:35563)
0925 21:56:02.805269 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.805272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.805282 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:02.805314 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.805316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.805329 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.805432 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:02.805585 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.806785 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.807940 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.808419 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.808894 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.808965 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.808990 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.809029 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":69,"threads_started":1}
I20260925 21:56:02.809230 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.805025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:39735 (local address 127.0.0.1:52866)
0925 21:56:02.805591 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:02.805600 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:02.805666 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:02.805831 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:02.805831 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:02.805833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:02.805877 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.805878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.807010 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.807011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.808091 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.808094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.809025 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.809026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.809078 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.809081 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:02.809125 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 21:56:02.809164 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":54,"threads_started":1}
I20260925 21:56:02.808755 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.804686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:45299 (local address 127.0.0.1:53076)
0925 21:56:02.804863 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:02.804907 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:02.804977 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:02.805286 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:02.805289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:02.805291 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:02.805361 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.805363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.806409 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.806410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.807519 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.807521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.808450 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.808451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.808539 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.808555 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 21:56:02.808623 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 21:56:02.808654 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":58,"threads_started":1}
I20260925 21:56:02.809674 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.805188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.2.70.195:39735)
0925 21:56:02.805374 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.805378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.805669 (+ 291us) server_negotiation.cc:407] Connection header received
0925 21:56:02.805731 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.805734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.805750 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.805827 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:02.805932 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.806946 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.808182 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.808876 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.808938 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.808995 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.809156 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.809484 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260925 21:56:02.809828 3580 tablet_service.cc:1467] Processing CreateTablet for tablet 30a6b43560124e9cb8c801566cc7ed72 (DEFAULT_TABLE table=test-table [id=5a88dec3d25440b8a68be1c50f4813ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:02.809968 3720 tablet_service.cc:1467] Processing CreateTablet for tablet 30a6b43560124e9cb8c801566cc7ed72 (DEFAULT_TABLE table=test-table [id=5a88dec3d25440b8a68be1c50f4813ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:02.810142 3580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30a6b43560124e9cb8c801566cc7ed72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:02.810314 3720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30a6b43560124e9cb8c801566cc7ed72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:02.810237 3860 tablet_service.cc:1467] Processing CreateTablet for tablet 30a6b43560124e9cb8c801566cc7ed72 (DEFAULT_TABLE table=test-table [id=5a88dec3d25440b8a68be1c50f4813ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:02.810510 3860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30a6b43560124e9cb8c801566cc7ed72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:02.813016 3949 tablet_bootstrap.cc:492] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d: Bootstrap starting.
I20260925 21:56:02.813532 3950 tablet_bootstrap.cc:492] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f: Bootstrap starting.
I20260925 21:56:02.813761 3949 tablet_bootstrap.cc:654] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:02.814100 3951 tablet_bootstrap.cc:492] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Bootstrap starting.
I20260925 21:56:02.814286 3949 log.cc:824] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:02.814443 3950 tablet_bootstrap.cc:654] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:02.814661 3951 tablet_bootstrap.cc:654] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:02.814782 3950 log.cc:824] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:02.814955 3951 log.cc:824] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:02.815474 3950 tablet_bootstrap.cc:492] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f: No bootstrap required, opened a new log
I20260925 21:56:02.815527 3950 ts_tablet_manager.cc:1397] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:56:02.815621 3951 tablet_bootstrap.cc:492] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46: No bootstrap required, opened a new log
I20260925 21:56:02.815678 3951 ts_tablet_manager.cc:1397] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:02.816233 3949 tablet_bootstrap.cc:492] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d: No bootstrap required, opened a new log
I20260925 21:56:02.816301 3949 ts_tablet_manager.cc:1397] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 21:56:02.817212 3951 raft_consensus.cc:348] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.817389 3951 raft_consensus.cc:374] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:02.817433 3951 raft_consensus.cc:729] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b026ab7c0d4de9a5c1138c8fbd0e46, State: Initialized, Role: FOLLOWER
I20260925 21:56:02.817544 3951 consensus_queue.cc:260] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.817512 3950 raft_consensus.cc:348] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.817664 3950 raft_consensus.cc:374] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:02.817695 3950 raft_consensus.cc:729] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38af88947c234739b31924fb1eef8a2f, State: Initialized, Role: FOLLOWER
I20260925 21:56:02.817800 3951 ts_tablet_manager.cc:1428] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:56:02.817818 3950 consensus_queue.cc:260] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.817914 3786 heartbeater.cc:503] Master 127.2.70.254:35175 was elected leader, sending a full tablet report...
I20260925 21:56:02.818066 3950 ts_tablet_manager.cc:1428] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:56:02.818133 3926 heartbeater.cc:503] Master 127.2.70.254:35175 was elected leader, sending a full tablet report...
I20260925 21:56:02.818172 3949 raft_consensus.cc:348] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.818320 3949 raft_consensus.cc:374] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:02.818361 3949 raft_consensus.cc:729] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121c9199ec25465d9a375a11a0d0568d, State: Initialized, Role: FOLLOWER
I20260925 21:56:02.818456 3949 consensus_queue.cc:260] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.818704 3949 ts_tablet_manager.cc:1428] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 21:56:02.818753 3646 heartbeater.cc:503] Master 127.2.70.254:35175 was elected leader, sending a full tablet report...
I20260925 21:56:02.860297 3956 raft_consensus.cc:482] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:02.860440 3956 raft_consensus.cc:504] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
W20260925 21:56:02.860566 3787 tablet.cc:2369] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:02.860870 3956 leader_election.cc:288] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84b026ab7c0d4de9a5c1138c8fbd0e46 (127.2.70.194:45299), 121c9199ec25465d9a375a11a0d0568d (127.2.70.193:35563)
I20260925 21:56:02.864701 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.861125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:44161 (local address 127.2.70.194:45299)
0925 21:56:02.861172 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.861174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.861186 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:56:02.861218 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.861219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.861252 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.861322 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:02.861497 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.862268 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.863750 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.864346 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.864410 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.864420 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.864585 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.864625 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 21:56:02.864728 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.861047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:45299 (local address 127.2.70.195:44161)
0925 21:56:02.861088 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:02.861101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:02.861146 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:02.861333 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:02.861334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:02.861336 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:02.861442 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.861444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.862352 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.862352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.863648 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.863652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.864420 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.864421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.864522 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.864526 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:02.864578 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 21:56:02.864628 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":98}
I20260925 21:56:02.864701 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.861266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:56913 (local address 127.2.70.193:35563)
0925 21:56:02.861305 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.861307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.861366 (+ 59us) server_negotiation.cc:407] Connection header received
0925 21:56:02.861428 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.861429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.861440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.861485 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:02.861612 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.862300 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.863754 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.864338 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.864412 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.864420 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.864591 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.864629 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 21:56:02.864733 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.861156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:35563 (local address 127.2.70.195:56913)
0925 21:56:02.861312 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:02.861320 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:02.861370 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:02.861504 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:02.861504 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:02.861506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:02.861557 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.861557 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.862353 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.862353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.863648 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:02.863652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.864421 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.864421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:02.864522 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.864526 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:02.864578 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 21:56:02.864628 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":30,"thread_start_us":120,"threads_started":1}
I20260925 21:56:02.865103 3740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a6b43560124e9cb8c801566cc7ed72" candidate_uuid: "38af88947c234739b31924fb1eef8a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" is_pre_election: true
I20260925 21:56:02.865228 3600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a6b43560124e9cb8c801566cc7ed72" candidate_uuid: "38af88947c234739b31924fb1eef8a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121c9199ec25465d9a375a11a0d0568d" is_pre_election: true
I20260925 21:56:02.865273 3740 raft_consensus.cc:2454] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38af88947c234739b31924fb1eef8a2f in term 0.
I20260925 21:56:02.865379 3600 raft_consensus.cc:2454] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38af88947c234739b31924fb1eef8a2f in term 0.
I20260925 21:56:02.865494 3814 leader_election.cc:302] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38af88947c234739b31924fb1eef8a2f, 84b026ab7c0d4de9a5c1138c8fbd0e46; no voters:
I20260925 21:56:02.865633 3956 raft_consensus.cc:2781] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:02.865690 3956 raft_consensus.cc:482] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:02.865713 3956 raft_consensus.cc:3037] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:02.866434 3956 raft_consensus.cc:504] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.866560 3956 leader_election.cc:288] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [CANDIDATE]: Term 1 election: Requested vote from peers 84b026ab7c0d4de9a5c1138c8fbd0e46 (127.2.70.194:45299), 121c9199ec25465d9a375a11a0d0568d (127.2.70.193:35563)
I20260925 21:56:02.866753 3740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a6b43560124e9cb8c801566cc7ed72" candidate_uuid: "38af88947c234739b31924fb1eef8a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46"
I20260925 21:56:02.866770 3600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a6b43560124e9cb8c801566cc7ed72" candidate_uuid: "38af88947c234739b31924fb1eef8a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121c9199ec25465d9a375a11a0d0568d"
I20260925 21:56:02.866840 3740 raft_consensus.cc:3037] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:02.866842 3600 raft_consensus.cc:3037] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:02.867547 3600 raft_consensus.cc:2454] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38af88947c234739b31924fb1eef8a2f in term 1.
I20260925 21:56:02.867555 3740 raft_consensus.cc:2454] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38af88947c234739b31924fb1eef8a2f in term 1.
I20260925 21:56:02.867714 3814 leader_election.cc:302] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121c9199ec25465d9a375a11a0d0568d, 38af88947c234739b31924fb1eef8a2f; no voters:
I20260925 21:56:02.867810 3956 raft_consensus.cc:2781] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:02.868003 3956 raft_consensus.cc:686] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 1 LEADER]: Becoming Leader. State: Replica: 38af88947c234739b31924fb1eef8a2f, State: Running, Role: LEADER
I20260925 21:56:02.868096 3956 consensus_queue.cc:237] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.868983 3443 catalog_manager.cc:5495] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 38af88947c234739b31924fb1eef8a2f (127.2.70.195). New cstate: current_term: 1 leader_uuid: "38af88947c234739b31924fb1eef8a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } health_report { overall_health: HEALTHY } } }
I20260925 21:56:02.883496 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:02.881106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.2.70.195:39735)
0925 21:56:02.881170 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:02.881172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:02.881181 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:02.881283 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:02.881284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:02.881306 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:02.881354 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:02.881460 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.881941 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.882704 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:02.882928 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:02.882992 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:02.883129 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:02.883230 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:02.883312 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:02.883327 (+ 15us) server_negotiation.cc:299] Negotiation successful
0925 21:56:02.883358 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 21:56:02.885675 3740 raft_consensus.cc:1261] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 1 FOLLOWER]: Refusing update from remote peer 38af88947c234739b31924fb1eef8a2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:02.885675 3600 raft_consensus.cc:1261] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d [term 1 FOLLOWER]: Refusing update from remote peer 38af88947c234739b31924fb1eef8a2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:02.886050 3956 consensus_queue.cc:1035] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:02.886209 3959 consensus_queue.cc:1035] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:02.889002 3964 mvcc.cc:205] Tried to move back new op lower bound from 7333369294376730624 to 7333369294308532224. Current Snapshot: MvccSnapshot[applied={T|T < 7333369294376730624}]
I20260925 21:56:02.889252 3963 mvcc.cc:205] Tried to move back new op lower bound from 7333369294376730624 to 7333369294308532224. Current Snapshot: MvccSnapshot[applied={T|T < 7333369294376730624}]
I20260925 21:56:02.891814 3442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:02.891971 3442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:02.892743 3442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765: Sending DeleteTablet for 3 replicas of tablet 30a6b43560124e9cb8c801566cc7ed72
I20260925 21:56:02.893091 3720 tablet_service.cc:1514] Processing DeleteTablet for tablet 30a6b43560124e9cb8c801566cc7ed72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:02 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:53076
I20260925 21:56:02.893136 3860 tablet_service.cc:1514] Processing DeleteTablet for tablet 30a6b43560124e9cb8c801566cc7ed72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:02 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:52866
I20260925 21:56:02.893121 3580 tablet_service.cc:1514] Processing DeleteTablet for tablet 30a6b43560124e9cb8c801566cc7ed72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:02 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:38196
I20260925 21:56:02.893325 3973 tablet_replica.cc:330] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f: stopping tablet replica
I20260925 21:56:02.893412 3973 raft_consensus.cc:2229] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:02.893500 3973 raft_consensus.cc:2258] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:02.893662 3974 tablet_replica.cc:330] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d: stopping tablet replica
I20260925 21:56:02.893823 3974 raft_consensus.cc:2229] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:02.893867 3972 tablet_replica.cc:330] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46: stopping tablet replica
I20260925 21:56:02.893848 3442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 21:56:02.893963 3972 raft_consensus.cc:2229] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 21:56:02.893996 3442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:02.894064 3972 raft_consensus.cc:2258] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:02.894212 3974 raft_consensus.cc:2258] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:02.894826 3973 ts_tablet_manager.cc:1905] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:02.895259 3974 ts_tablet_manager.cc:1905] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:02.895749 3580 tablet_service.cc:1467] Processing CreateTablet for tablet a1a63c29ae3c4dde9c11459c388bf873 (DEFAULT_TABLE table=test-table [id=e4cec0ff685f4440985f69f5a101ec41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:02.895884 3580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a63c29ae3c4dde9c11459c388bf873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:02.896546 3720 tablet_service.cc:1467] Processing CreateTablet for tablet a1a63c29ae3c4dde9c11459c388bf873 (DEFAULT_TABLE table=test-table [id=e4cec0ff685f4440985f69f5a101ec41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:02.896647 3720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a63c29ae3c4dde9c11459c388bf873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:02.897089 3973 ts_tablet_manager.cc:1918] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:02.897135 3949 tablet_bootstrap.cc:492] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d: Bootstrap starting.
I20260925 21:56:02.897150 3973 log.cc:1196] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/30a6b43560124e9cb8c801566cc7ed72
I20260925 21:56:02.897449 3973 ts_tablet_manager.cc:1939] T 30a6b43560124e9cb8c801566cc7ed72 P 38af88947c234739b31924fb1eef8a2f: Deleting consensus metadata
I20260925 21:56:02.897535 3949 tablet_bootstrap.cc:654] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:02.897856 3951 tablet_bootstrap.cc:492] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Bootstrap starting.
I20260925 21:56:02.897907 3974 ts_tablet_manager.cc:1918] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:02.897971 3974 log.cc:1196] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/wal/wals/30a6b43560124e9cb8c801566cc7ed72
I20260925 21:56:02.898080 3860 tablet_service.cc:1467] Processing CreateTablet for tablet a1a63c29ae3c4dde9c11459c388bf873 (DEFAULT_TABLE table=test-table [id=e4cec0ff685f4440985f69f5a101ec41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:02.898162 3860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a63c29ae3c4dde9c11459c388bf873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:02.898226 3951 tablet_bootstrap.cc:654] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:02.898327 3974 ts_tablet_manager.cc:1939] T 30a6b43560124e9cb8c801566cc7ed72 P 121c9199ec25465d9a375a11a0d0568d: Deleting consensus metadata
I20260925 21:56:02.898779 3949 tablet_bootstrap.cc:492] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d: No bootstrap required, opened a new log
I20260925 21:56:02.898818 3949 ts_tablet_manager.cc:1397] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:02.898943 3951 tablet_bootstrap.cc:492] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46: No bootstrap required, opened a new log
I20260925 21:56:02.898979 3951 ts_tablet_manager.cc:1397] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:02.898967 3949 raft_consensus.cc:348] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.899039 3949 raft_consensus.cc:374] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:02.899060 3949 raft_consensus.cc:729] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121c9199ec25465d9a375a11a0d0568d, State: Initialized, Role: FOLLOWER
I20260925 21:56:02.899106 3949 consensus_queue.cc:260] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.899106 3951 raft_consensus.cc:348] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.899158 3951 raft_consensus.cc:374] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:02.899175 3951 raft_consensus.cc:729] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b026ab7c0d4de9a5c1138c8fbd0e46, State: Initialized, Role: FOLLOWER
I20260925 21:56:02.899194 3949 ts_tablet_manager.cc:1428] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:02.899209 3951 consensus_queue.cc:260] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.899282 3951 ts_tablet_manager.cc:1428] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:02.899374 3950 tablet_bootstrap.cc:492] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f: Bootstrap starting.
I20260925 21:56:02.899696 3430 catalog_manager.cc:4841] TS 121c9199ec25465d9a375a11a0d0568d (127.2.70.193:35563): tablet 30a6b43560124e9cb8c801566cc7ed72 (table test-table [id=5a88dec3d25440b8a68be1c50f4813ba]) successfully deleted
I20260925 21:56:02.899749 3950 tablet_bootstrap.cc:654] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:02.899804 3430 catalog_manager.cc:4841] TS 38af88947c234739b31924fb1eef8a2f (127.2.70.195:39735): tablet 30a6b43560124e9cb8c801566cc7ed72 (table test-table [id=5a88dec3d25440b8a68be1c50f4813ba]) successfully deleted
I20260925 21:56:02.900236 3972 ts_tablet_manager.cc:1905] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:02.901700 3972 ts_tablet_manager.cc:1918] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:02.901767 3972 log.cc:1196] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/30a6b43560124e9cb8c801566cc7ed72
I20260925 21:56:02.901965 3972 ts_tablet_manager.cc:1939] T 30a6b43560124e9cb8c801566cc7ed72 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Deleting consensus metadata
I20260925 21:56:02.902319 3430 catalog_manager.cc:4841] TS 84b026ab7c0d4de9a5c1138c8fbd0e46 (127.2.70.194:45299): tablet 30a6b43560124e9cb8c801566cc7ed72 (table test-table [id=5a88dec3d25440b8a68be1c50f4813ba]) successfully deleted
I20260925 21:56:02.904273 3950 tablet_bootstrap.cc:492] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f: No bootstrap required, opened a new log
I20260925 21:56:02.904321 3950 ts_tablet_manager.cc:1397] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 21:56:02.904446 3950 raft_consensus.cc:348] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.904505 3950 raft_consensus.cc:374] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:02.904516 3950 raft_consensus.cc:729] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38af88947c234739b31924fb1eef8a2f, State: Initialized, Role: FOLLOWER
I20260925 21:56:02.904561 3950 consensus_queue.cc:260] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:02.904652 3950 ts_tablet_manager.cc:1428] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260925 21:56:02.957324 3647 tablet.cc:2369] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 21:56:03.009490 3927 tablet.cc:2369] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:03.044869 3955 raft_consensus.cc:482] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:03.044967 3955 raft_consensus.cc:504] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:03.045233 3955 leader_election.cc:288] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 121c9199ec25465d9a375a11a0d0568d (127.2.70.193:35563), 38af88947c234739b31924fb1eef8a2f (127.2.70.195:39735)
I20260925 21:56:03.048075 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.045476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:39707 (local address 127.2.70.193:35563)
0925 21:56:03.045524 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:03.045526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:03.045536 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:03.045562 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:03.045563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:03.045571 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:03.045631 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:03.045800 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.046360 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.047385 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.047731 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.047796 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.047805 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:03.047972 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 21:56:03.048008 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 21:56:03.048092 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.045402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:35563 (local address 127.2.70.194:39707)
0925 21:56:03.045453 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:03.045466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:03.045511 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:03.045651 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:03.045653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:03.045656 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:03.045728 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:03.045729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.046442 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.046443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:03.047292 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:03.047295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.047805 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.047807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:03.047910 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.047913 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:03.047958 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 21:56:03.047997 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":37}
I20260925 21:56:03.048089 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.045650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:58761 (local address 127.2.70.195:39735)
0925 21:56:03.045750 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:03.045752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:03.045758 (+ 6us) server_negotiation.cc:407] Connection header received
0925 21:56:03.045781 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:03.045781 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:03.045789 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:03.045826 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:03.045957 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.046400 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.047384 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.047740 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.047800 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.047809 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:03.047988 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 21:56:03.048024 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 21:56:03.048115 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.045507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:39735 (local address 127.2.70.194:58761)
0925 21:56:03.045630 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:03.045639 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:03.045678 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:03.045836 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:03.045836 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:03.045838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:03.045899 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:03.045901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.046485 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.046486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:03.047298 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:03.047300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.047851 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.047852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:03.047924 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.047927 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:03.047974 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 21:56:03.048011 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260925 21:56:03.048388 3600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a63c29ae3c4dde9c11459c388bf873" candidate_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121c9199ec25465d9a375a11a0d0568d" is_pre_election: true
I20260925 21:56:03.048483 3600 raft_consensus.cc:2454] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84b026ab7c0d4de9a5c1138c8fbd0e46 in term 0.
I20260925 21:56:03.048560 3880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a63c29ae3c4dde9c11459c388bf873" candidate_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38af88947c234739b31924fb1eef8a2f" is_pre_election: true
I20260925 21:56:03.048660 3674 leader_election.cc:302] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121c9199ec25465d9a375a11a0d0568d, 84b026ab7c0d4de9a5c1138c8fbd0e46; no voters:
I20260925 21:56:03.048703 3880 raft_consensus.cc:2454] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84b026ab7c0d4de9a5c1138c8fbd0e46 in term 0.
I20260925 21:56:03.048801 3955 raft_consensus.cc:2781] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:03.048841 3955 raft_consensus.cc:482] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:03.048866 3955 raft_consensus.cc:3037] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:03.049541 3955 raft_consensus.cc:504] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:03.049656 3955 leader_election.cc:288] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [CANDIDATE]: Term 1 election: Requested vote from peers 121c9199ec25465d9a375a11a0d0568d (127.2.70.193:35563), 38af88947c234739b31924fb1eef8a2f (127.2.70.195:39735)
I20260925 21:56:03.049832 3600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a63c29ae3c4dde9c11459c388bf873" candidate_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121c9199ec25465d9a375a11a0d0568d"
I20260925 21:56:03.049914 3600 raft_consensus.cc:3037] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:03.049896 3880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a63c29ae3c4dde9c11459c388bf873" candidate_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38af88947c234739b31924fb1eef8a2f"
I20260925 21:56:03.049953 3880 raft_consensus.cc:3037] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:03.050642 3600 raft_consensus.cc:2454] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84b026ab7c0d4de9a5c1138c8fbd0e46 in term 1.
I20260925 21:56:03.050673 3880 raft_consensus.cc:2454] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84b026ab7c0d4de9a5c1138c8fbd0e46 in term 1.
I20260925 21:56:03.050836 3674 leader_election.cc:302] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38af88947c234739b31924fb1eef8a2f, 84b026ab7c0d4de9a5c1138c8fbd0e46; no voters:
I20260925 21:56:03.050931 3955 raft_consensus.cc:2781] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:03.051071 3955 raft_consensus.cc:686] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 1 LEADER]: Becoming Leader. State: Replica: 84b026ab7c0d4de9a5c1138c8fbd0e46, State: Running, Role: LEADER
I20260925 21:56:03.051129 3955 consensus_queue.cc:237] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:03.051810 3442 catalog_manager.cc:5495] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84b026ab7c0d4de9a5c1138c8fbd0e46 (127.2.70.194). New cstate: current_term: 1 leader_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } health_report { overall_health: UNKNOWN } } }
Sep 25 21:56:03 dist-test-slave-2g0f krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:03.107573 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.100782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.2.70.254:35175)
0925 21:56:03.100840 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:03.100842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:03.100851 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:03.100915 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:03.100915 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:03.100924 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:03.100969 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:03.101402 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.101896 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.102609 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.102820 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.102887 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.104931 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:03.104932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:03.104932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:03.104939 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:03.106450 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:03.106866 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:03.106867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:03.106867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:03.106928 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:03.107119 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:03.107120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:03.107120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:03.107232 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:03.107279 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:03.107434 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 21:56:03.107503 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 21:56:03.107698 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.100623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35175 (local address 127.0.0.1:45646)
0925 21:56:03.100783 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:03.100795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:03.100862 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:03.100983 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:03.100984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:03.101298 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:03.101348 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:03.101350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.101961 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.101962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:03.102509 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:03.102510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.102896 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.102897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:03.102935 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.103466 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:03.103470 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:03.104826 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:03.106544 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:03.106552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:03.106554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:03.106793 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:03.107004 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:03.107005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:03.107005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:03.107068 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:03.107298 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:03.107305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:03.107355 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 21:56:03.107413 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 21:56:03.107603 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":68,"thread_start_us":75,"threads_started":1}
I20260925 21:56:03.109750 3442 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:45646:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 21:56:03.109931 3442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:03.111366 3720 tablet_service.cc:1467] Processing CreateTablet for tablet 18771abdfa8342c8ada21568f49c9d4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c5a1f5c542a4327bdfb177563b779c2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 21:56:03.111442 3860 tablet_service.cc:1467] Processing CreateTablet for tablet 18771abdfa8342c8ada21568f49c9d4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c5a1f5c542a4327bdfb177563b779c2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 21:56:03.111436 3580 tablet_service.cc:1467] Processing CreateTablet for tablet 18771abdfa8342c8ada21568f49c9d4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c5a1f5c542a4327bdfb177563b779c2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 21:56:03.111557 3720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18771abdfa8342c8ada21568f49c9d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:03.111599 3580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18771abdfa8342c8ada21568f49c9d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:03.111601 3860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18771abdfa8342c8ada21568f49c9d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:03.112931 3949 tablet_bootstrap.cc:492] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d: Bootstrap starting.
I20260925 21:56:03.113016 3950 tablet_bootstrap.cc:492] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f: Bootstrap starting.
I20260925 21:56:03.113044 3951 tablet_bootstrap.cc:492] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: Bootstrap starting.
I20260925 21:56:03.113355 3950 tablet_bootstrap.cc:654] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:03.113355 3949 tablet_bootstrap.cc:654] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:03.113417 3951 tablet_bootstrap.cc:654] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:03.113890 3950 tablet_bootstrap.cc:492] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f: No bootstrap required, opened a new log
I20260925 21:56:03.113941 3950 ts_tablet_manager.cc:1397] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:03.114051 3950 raft_consensus.cc:348] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:03.114101 3950 raft_consensus.cc:374] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:03.114112 3950 raft_consensus.cc:729] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38af88947c234739b31924fb1eef8a2f, State: Initialized, Role: FOLLOWER
I20260925 21:56:03.114112 3949 tablet_bootstrap.cc:492] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d: No bootstrap required, opened a new log
I20260925 21:56:03.114154 3949 ts_tablet_manager.cc:1397] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:03.114146 3950 consensus_queue.cc:260] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:03.114239 3950 ts_tablet_manager.cc:1428] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:03.114256 3956 tablet_replica.cc:439] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } } }
I20260925 21:56:03.114279 3949 raft_consensus.cc:348] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:03.114331 3956 tablet_replica.cc:442] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:03.114362 3949 raft_consensus.cc:374] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:03.114383 3949 raft_consensus.cc:729] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121c9199ec25465d9a375a11a0d0568d, State: Initialized, Role: FOLLOWER
I20260925 21:56:03.114419 3949 consensus_queue.cc:260] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:03.114511 3949 ts_tablet_manager.cc:1428] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:03.114562 3951 tablet_bootstrap.cc:492] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: No bootstrap required, opened a new log
I20260925 21:56:03.114604 3951 ts_tablet_manager.cc:1397] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:03.114660 3957 tablet_replica.cc:439] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } } }
I20260925 21:56:03.114760 3957 tablet_replica.cc:442] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:03.114763 3951 raft_consensus.cc:348] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:03.114828 3951 raft_consensus.cc:374] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:03.114847 3951 raft_consensus.cc:729] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b026ab7c0d4de9a5c1138c8fbd0e46, State: Initialized, Role: FOLLOWER
I20260925 21:56:03.114886 3951 consensus_queue.cc:260] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:03.114979 3951 ts_tablet_manager.cc:1428] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:03.115006 3955 tablet_replica.cc:439] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } } }
I20260925 21:56:03.115063 3955 tablet_replica.cc:442] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:03.206362 3955 raft_consensus.cc:482] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:03.206453 3955 raft_consensus.cc:504] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:03.206583 3955 leader_election.cc:288] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 121c9199ec25465d9a375a11a0d0568d (127.2.70.193:35563), 38af88947c234739b31924fb1eef8a2f (127.2.70.195:39735)
I20260925 21:56:03.206815 3600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18771abdfa8342c8ada21568f49c9d4a" candidate_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121c9199ec25465d9a375a11a0d0568d" is_pre_election: true
I20260925 21:56:03.206835 3880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18771abdfa8342c8ada21568f49c9d4a" candidate_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38af88947c234739b31924fb1eef8a2f" is_pre_election: true
I20260925 21:56:03.206907 3600 raft_consensus.cc:2454] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84b026ab7c0d4de9a5c1138c8fbd0e46 in term 0.
I20260925 21:56:03.206930 3880 raft_consensus.cc:2454] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84b026ab7c0d4de9a5c1138c8fbd0e46 in term 0.
I20260925 21:56:03.207059 3674 leader_election.cc:302] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121c9199ec25465d9a375a11a0d0568d, 84b026ab7c0d4de9a5c1138c8fbd0e46; no voters:
I20260925 21:56:03.207163 3955 raft_consensus.cc:2781] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:03.207227 3955 raft_consensus.cc:482] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:03.207242 3955 raft_consensus.cc:3037] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:03.207923 3955 raft_consensus.cc:504] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:03.208024 3955 leader_election.cc:288] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [CANDIDATE]: Term 1 election: Requested vote from peers 121c9199ec25465d9a375a11a0d0568d (127.2.70.193:35563), 38af88947c234739b31924fb1eef8a2f (127.2.70.195:39735)
I20260925 21:56:03.208165 3600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18771abdfa8342c8ada21568f49c9d4a" candidate_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121c9199ec25465d9a375a11a0d0568d"
I20260925 21:56:03.208235 3600 raft_consensus.cc:3037] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:03.208247 3880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18771abdfa8342c8ada21568f49c9d4a" candidate_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38af88947c234739b31924fb1eef8a2f"
I20260925 21:56:03.208324 3880 raft_consensus.cc:3037] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:03.209048 3600 raft_consensus.cc:2454] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84b026ab7c0d4de9a5c1138c8fbd0e46 in term 1.
I20260925 21:56:03.209151 3880 raft_consensus.cc:2454] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84b026ab7c0d4de9a5c1138c8fbd0e46 in term 1.
I20260925 21:56:03.209208 3674 leader_election.cc:302] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121c9199ec25465d9a375a11a0d0568d, 84b026ab7c0d4de9a5c1138c8fbd0e46; no voters:
I20260925 21:56:03.209319 3955 raft_consensus.cc:2781] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:03.209374 3955 raft_consensus.cc:686] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 1 LEADER]: Becoming Leader. State: Replica: 84b026ab7c0d4de9a5c1138c8fbd0e46, State: Running, Role: LEADER
I20260925 21:56:03.209424 3955 consensus_queue.cc:237] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } }
I20260925 21:56:03.209726 3979 tablet_replica.cc:439] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: TxnStatusTablet state changed. Reason: New leader 84b026ab7c0d4de9a5c1138c8fbd0e46. Latest consensus state: current_term: 1 leader_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } } }
I20260925 21:56:03.209817 3979 tablet_replica.cc:442] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: This TxnStatusTablet replica's current role is: LEADER
I20260925 21:56:03.210028 3991 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 21:56:03.210016 3443 catalog_manager.cc:5495] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84b026ab7c0d4de9a5c1138c8fbd0e46 (127.2.70.194). New cstate: current_term: 1 leader_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } health_report { overall_health: UNKNOWN } } }
I20260925 21:56:03.210155 3991 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 21:56:03.233562 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.231046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:45299 (local address 127.0.0.1:53080)
0925 21:56:03.231102 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:03.231114 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:03.231163 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:03.231345 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:03.231347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:03.231349 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 21:56:03.231410 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:03.231413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.232126 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.232127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:03.232720 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:03.232723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.233057 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.233058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:03.233127 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.233146 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 21:56:03.233391 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 21:56:03.233399 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 21:56:03.233442 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 21:56:03.233494 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 21:56:03.233566 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.231108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.2.70.194:45299)
0925 21:56:03.231156 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:03.231158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:03.231168 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:03.231250 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:03.231251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:03.231275 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:03.231323 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:03.231467 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.232052 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.232798 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.232999 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.233054 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.233229 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:03.233338 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:03.233398 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:03.233445 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 21:56:03.233465 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 21:56:03.234748 3600 raft_consensus.cc:1261] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d [term 1 FOLLOWER]: Refusing update from remote peer 84b026ab7c0d4de9a5c1138c8fbd0e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:03.234781 3880 raft_consensus.cc:1261] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f [term 1 FOLLOWER]: Refusing update from remote peer 84b026ab7c0d4de9a5c1138c8fbd0e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:03.235029 3979 consensus_queue.cc:1035] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:03.235175 3955 consensus_queue.cc:1035] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:03.235975 3956 tablet_replica.cc:439] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f: TxnStatusTablet state changed. Reason: New leader 84b026ab7c0d4de9a5c1138c8fbd0e46. Latest consensus state: current_term: 1 leader_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } } }
I20260925 21:56:03.236063 3956 tablet_replica.cc:442] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:03.236610 3957 tablet_replica.cc:439] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d: TxnStatusTablet state changed. Reason: New leader 84b026ab7c0d4de9a5c1138c8fbd0e46. Latest consensus state: current_term: 1 leader_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } } }
I20260925 21:56:03.236717 3957 tablet_replica.cc:442] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:03.236690 3979 tablet_replica.cc:439] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } } }
I20260925 21:56:03.236754 3979 tablet_replica.cc:442] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: This TxnStatusTablet replica's current role is: LEADER
I20260925 21:56:03.237313 3979 tablet_replica.cc:439] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } } }
I20260925 21:56:03.237382 3979 tablet_replica.cc:442] T 18771abdfa8342c8ada21568f49c9d4a P 84b026ab7c0d4de9a5c1138c8fbd0e46: This TxnStatusTablet replica's current role is: LEADER
I20260925 21:56:03.237541 3957 tablet_replica.cc:439] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } } }
I20260925 21:56:03.237619 3957 tablet_replica.cc:442] T 18771abdfa8342c8ada21568f49c9d4a P 121c9199ec25465d9a375a11a0d0568d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:03.238090 3956 tablet_replica.cc:439] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 } } peers { permanent_uuid: "84b026ab7c0d4de9a5c1138c8fbd0e46" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 45299 } } peers { permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 } } }
I20260925 21:56:03.238169 3956 tablet_replica.cc:442] T 18771abdfa8342c8ada21568f49c9d4a P 38af88947c234739b31924fb1eef8a2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:03.238209 3962 mvcc.cc:205] Tried to move back new op lower bound from 7333369295807365120 to 7333369295706144768. Current Snapshot: MvccSnapshot[applied={T|T < 7333369295807365120 or (T in {7333369295807365120})}]
I20260925 21:56:03.241219 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.239099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.2.70.194:45299)
0925 21:56:03.239134 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:03.239136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:03.239144 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:03.239237 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:03.239238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:03.239248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:03.239298 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:03.239403 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.239936 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.240628 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.240826 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.240879 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.241001 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:03.241035 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:03.241077 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:03.241130 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 21:56:03.241149 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 21:56:03.246672 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.244380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:35557 (local address 127.2.70.194:45299)
0925 21:56:03.244424 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:03.244426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:03.244556 (+ 130us) server_negotiation.cc:407] Connection header received
0925 21:56:03.244602 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:03.244603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:03.244612 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:03.244659 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:03.244799 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.245251 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.246021 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.246207 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.246259 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.246432 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:03.246465 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:03.246508 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:03.246562 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 21:56:03.246577 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":23}
I20260925 21:56:03.246688 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.244324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:45299 (local address 127.2.70.194:35557)
0925 21:56:03.244491 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:03.244500 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:03.244556 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:03.244662 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:03.244663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:03.244664 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 21:56:03.244740 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:03.244742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.245318 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.245319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:03.245944 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:03.245946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.246289 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.246290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:03.246351 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.246365 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 21:56:03.246516 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 21:56:03.246517 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 21:56:03.246552 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 21:56:03.246592 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
I20260925 21:56:03.248914 3600 raft_consensus.cc:1261] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d [term 1 FOLLOWER]: Refusing update from remote peer 84b026ab7c0d4de9a5c1138c8fbd0e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:03.248908 3880 raft_consensus.cc:1261] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f [term 1 FOLLOWER]: Refusing update from remote peer 84b026ab7c0d4de9a5c1138c8fbd0e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:03.249131 3955 consensus_queue.cc:1035] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38af88947c234739b31924fb1eef8a2f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:03.249253 3979 consensus_queue.cc:1035] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "121c9199ec25465d9a375a11a0d0568d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 35563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:03.263319 3443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:03.263445 3443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:03.264158 3443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2333b7aa326f4987b46c1e1f7a400765: Sending DeleteTablet for 3 replicas of tablet a1a63c29ae3c4dde9c11459c388bf873
I20260925 21:56:03.264408 3580 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a63c29ae3c4dde9c11459c388bf873 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:03 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:38196
I20260925 21:56:03.264436 3719 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a63c29ae3c4dde9c11459c388bf873 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:03 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:53076
I20260925 21:56:03.264483 3860 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a63c29ae3c4dde9c11459c388bf873 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:03 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:52866
I20260925 21:56:03.264555 3972 tablet_replica.cc:330] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46: stopping tablet replica
I20260925 21:56:03.264595 3974 tablet_replica.cc:330] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d: stopping tablet replica
I20260925 21:56:03.264600 3973 tablet_replica.cc:330] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f: stopping tablet replica
I20260925 21:56:03.264621 3972 raft_consensus.cc:2229] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:03.264643 3974 raft_consensus.cc:2229] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:03.264679 3974 raft_consensus.cc:2258] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:03.264680 3973 raft_consensus.cc:2229] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:03.264726 3972 raft_consensus.cc:2258] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:03.264736 3973 raft_consensus.cc:2258] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:03.265069 3972 ts_tablet_manager.cc:1905] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:03.265065 3974 ts_tablet_manager.cc:1905] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:03.265566 3973 ts_tablet_manager.cc:1905] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:03.266767 3974 ts_tablet_manager.cc:1918] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:03.266834 3974 log.cc:1196] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-0/wal/wals/a1a63c29ae3c4dde9c11459c388bf873
I20260925 21:56:03.266850 3972 ts_tablet_manager.cc:1918] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:03.266901 3972 log.cc:1196] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/a1a63c29ae3c4dde9c11459c388bf873
I20260925 21:56:03.267015 3973 ts_tablet_manager.cc:1918] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:03.267059 3973 log.cc:1196] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/a1a63c29ae3c4dde9c11459c388bf873
I20260925 21:56:03.267067 3974 ts_tablet_manager.cc:1939] T a1a63c29ae3c4dde9c11459c388bf873 P 121c9199ec25465d9a375a11a0d0568d: Deleting consensus metadata
I20260925 21:56:03.267099 3972 ts_tablet_manager.cc:1939] T a1a63c29ae3c4dde9c11459c388bf873 P 84b026ab7c0d4de9a5c1138c8fbd0e46: Deleting consensus metadata
I20260925 21:56:03.267228 3973 ts_tablet_manager.cc:1939] T a1a63c29ae3c4dde9c11459c388bf873 P 38af88947c234739b31924fb1eef8a2f: Deleting consensus metadata
I20260925 21:56:03.267472 3430 catalog_manager.cc:4841] TS 84b026ab7c0d4de9a5c1138c8fbd0e46 (127.2.70.194:45299): tablet a1a63c29ae3c4dde9c11459c388bf873 (table test-table [id=e4cec0ff685f4440985f69f5a101ec41]) successfully deleted
I20260925 21:56:03.267547 3430 catalog_manager.cc:4841] TS 121c9199ec25465d9a375a11a0d0568d (127.2.70.193:35563): tablet a1a63c29ae3c4dde9c11459c388bf873 (table test-table [id=e4cec0ff685f4440985f69f5a101ec41]) successfully deleted
I20260925 21:56:03.267782 3430 catalog_manager.cc:4841] TS 38af88947c234739b31924fb1eef8a2f (127.2.70.195:39735): tablet a1a63c29ae3c4dde9c11459c388bf873 (table test-table [id=e4cec0ff685f4440985f69f5a101ec41]) successfully deleted
Sep 25 21:56:03 dist-test-slave-2g0f krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.193@KRBTEST.COM
I20260925 21:56:03.276537 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.267604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.2.70.193:35563)
0925 21:56:03.267646 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:03.267648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:03.267800 (+ 152us) server_negotiation.cc:407] Connection header received
0925 21:56:03.267923 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:03.267924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:03.267944 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:03.267973 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:03.268700 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.269182 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.270249 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.270435 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.270498 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.273734 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:03.273745 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:03.273745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:03.273753 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:03.275336 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:03.275760 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:03.275763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:03.275764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:03.275871 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:03.276046 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:03.276046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:03.276047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:03.276196 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:03.276302 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:03.276395 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 21:56:03.276466 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 21:56:03.276808 3640 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38212
I20260925 21:56:03.284448 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:03.277745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.2.70.254:35175)
0925 21:56:03.277804 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:03.277806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:03.277915 (+ 109us) server_negotiation.cc:407] Connection header received
0925 21:56:03.277977 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:03.277977 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:03.277987 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:03.278029 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:03.278614 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.279204 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.280242 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:03.280438 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:03.280499 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:03.281890 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:03.281891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:03.281891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:03.281899 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:03.283238 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:03.283651 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:03.283652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:03.283652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:03.283724 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:03.284005 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:03.284006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:03.284007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:03.284126 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:03.284212 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:03.284318 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 21:56:03.284380 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 21:56:03.284729 3443 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45658
I20260925 21:56:03.285418 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 3515
I20260925 21:56:03.289935 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 3655
I20260925 21:56:03.294575 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 3795
I20260925 21:56:03.298857 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 3411
2026-09-25T21:56:03Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2248 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:03 dist-test-slave-2g0f krb5kdc[4010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:03 dist-test-slave-2g0f krb5kdc[4010](info): set up 2 sockets
Sep 25 21:56:03 dist-test-slave-2g0f krb5kdc[4010](info): commencing operation
krb5kdc: starting...
W20260925 21:56:04.263411 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:04 dist-test-slave-2g0f krb5kdc[4010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:56:04.382323 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:36933
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:36933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:04.464339 4023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:04.464569 4023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:04.464601 4023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:04.465804 4023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:04.465847 4023 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 21:56:04.465864 4023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:04.465880 4023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:04.465895 4023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:04.465921 4023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:04.467451 4023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:36933
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:36933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:04.467752 4023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:04.468000 4023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:04.472851 4031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:04.472851 4029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:04.472873 4028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:04.473186 4023 server_base.cc:1034] running on GCE node
I20260925 21:56:04.473435 4023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:04.473747 4023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:04.474896 4023 hybrid_clock.cc:648] HybridClock initialized: now 1790373364474877 us; error 34 us; skew 500 ppm
Sep 25 21:56:04 dist-test-slave-2g0f krb5kdc[4010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:04.477407 4023 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:56:04.477874 4023 webserver.cc:460] Webserver started at http://127.2.70.254:42497/ using document root <none> and password file <none>
I20260925 21:56:04.478112 4023 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:04.478169 4023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:04.478294 4023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:04.479374 4023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "33bf6a147986440d8890fa202d261337"
format_stamp: "Formatted at 2026-09-25 21:56:04 on dist-test-slave-2g0f"
server_key: "0883dcb6a8dd6b3374af5d2b52ba2b47"
server_key_iv: "438215981496c1eb503251aa930d0bef"
server_key_version: "encryptionkey@0"
I20260925 21:56:04.479700 4023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "33bf6a147986440d8890fa202d261337"
format_stamp: "Formatted at 2026-09-25 21:56:04 on dist-test-slave-2g0f"
server_key: "0883dcb6a8dd6b3374af5d2b52ba2b47"
server_key_iv: "438215981496c1eb503251aa930d0bef"
server_key_version: "encryptionkey@0"
I20260925 21:56:04.481166 4023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:04.482117 4038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:04.482347 4023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:04.482450 4023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "33bf6a147986440d8890fa202d261337"
format_stamp: "Formatted at 2026-09-25 21:56:04 on dist-test-slave-2g0f"
server_key: "0883dcb6a8dd6b3374af5d2b52ba2b47"
server_key_iv: "438215981496c1eb503251aa930d0bef"
server_key_version: "encryptionkey@0"
I20260925 21:56:04.482527 4023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:04.496471 4023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:04.498698 4023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:04.498822 4023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:04.503500 4023 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:36933
I20260925 21:56:04.503552 4100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:36933 every 8 connection(s)
I20260925 21:56:04.503917 4023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:04.504590 4101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:04.506937 4101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337: Bootstrap starting.
I20260925 21:56:04.507144 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 4023
I20260925 21:56:04.507231 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:04.507380 2331 external_mini_cluster.cc:1428] Setting key 22a9f69c82f741195e8577017890016d
I20260925 21:56:04.507640 4101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:04.507987 4101 log.cc:824] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:04.508688 4101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337: No bootstrap required, opened a new log
I20260925 21:56:04.510062 4101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bf6a147986440d8890fa202d261337" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 36933 } }
I20260925 21:56:04.510211 4101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:04.510244 4101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33bf6a147986440d8890fa202d261337, State: Initialized, Role: FOLLOWER
I20260925 21:56:04.510388 4101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bf6a147986440d8890fa202d261337" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 36933 } }
I20260925 21:56:04.510469 4101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:04.510502 4101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:04.510578 4101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:04.511149 4101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bf6a147986440d8890fa202d261337" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 36933 } }
I20260925 21:56:04.511248 4101 leader_election.cc:302] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33bf6a147986440d8890fa202d261337; no voters:
I20260925 21:56:04.511451 4101 leader_election.cc:288] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:04.511493 4106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:04.511626 4106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [term 1 LEADER]: Becoming Leader. State: Replica: 33bf6a147986440d8890fa202d261337, State: Running, Role: LEADER
I20260925 21:56:04.511713 4106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bf6a147986440d8890fa202d261337" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 36933 } }
I20260925 21:56:04.511755 4101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:04.512054 4106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33bf6a147986440d8890fa202d261337. Latest consensus state: current_term: 1 leader_uuid: "33bf6a147986440d8890fa202d261337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bf6a147986440d8890fa202d261337" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 36933 } } }
I20260925 21:56:04.512138 4106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:04.512086 4108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33bf6a147986440d8890fa202d261337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bf6a147986440d8890fa202d261337" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 36933 } } }
I20260925 21:56:04.512323 4108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:04.512331 4111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:04.513262 4111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 21:56:04 dist-test-slave-2g0f krb5kdc[4010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:04.515112 4111 catalog_manager.cc:1347] Generated new cluster ID: 38c8cc1acef24aeaa45018afad7f32cf
I20260925 21:56:04.515177 4111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:04.519080 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:04.507919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48646 (local address 127.2.70.254:36933)
0925 21:56:04.508087 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:04.508098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:04.508130 (+ 32us) server_negotiation.cc:407] Connection header received
0925 21:56:04.508610 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:04.508617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:04.508670 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:04.508812 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:04.509412 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.510791 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.511689 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.512353 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.512790 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:04.515067 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:04.515072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:04.515072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:04.515084 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:04.517103 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:04.517535 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:04.517538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:04.517539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:04.517666 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:04.517857 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:04.517858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:04.517859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:04.518194 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:04.518653 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:04.518725 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 21:56:04.518868 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260925 21:56:04.530237 4111 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:04.531011 4111 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:04.544926 4111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337: Generated new TSK 0
I20260925 21:56:04.545168 4111 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:56:04.575460 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:36933
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:04.657480 4127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:04.657707 4127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:04.657732 4127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:04.658937 4127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:04.658987 4127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:04.659008 4127 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 21:56:04.659017 4127 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 21:56:04.659034 4127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:04.660559 4127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.254:36933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:04.660794 4127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:04.661019 4127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:04.666257 4135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:04.666229 4132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:04.666292 4127 server_base.cc:1034] running on GCE node
W20260925 21:56:04.666227 4133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:04.666672 4127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:04.666895 4127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:04.668027 4127 hybrid_clock.cc:648] HybridClock initialized: now 1790373364668013 us; error 29 us; skew 500 ppm
Sep 25 21:56:04 dist-test-slave-2g0f krb5kdc[4010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:04.670523 4127 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:56:04.670954 4127 webserver.cc:460] Webserver started at http://127.2.70.193:36909/ using document root <none> and password file <none>
I20260925 21:56:04.671137 4127 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:04.671180 4127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:04.671272 4127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:04.672339 4127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "f2c83d207994493b93f92d1e3e420271"
format_stamp: "Formatted at 2026-09-25 21:56:04 on dist-test-slave-2g0f"
server_key: "1a2090da3e9d4338096d0ca56f8e3c75"
server_key_iv: "39295a11514f0cfd8d335f12e3e4dc83"
server_key_version: "encryptionkey@0"
I20260925 21:56:04.672643 4127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "f2c83d207994493b93f92d1e3e420271"
format_stamp: "Formatted at 2026-09-25 21:56:04 on dist-test-slave-2g0f"
server_key: "1a2090da3e9d4338096d0ca56f8e3c75"
server_key_iv: "39295a11514f0cfd8d335f12e3e4dc83"
server_key_version: "encryptionkey@0"
I20260925 21:56:04.673883 4127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:04.674525 4142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:04.674744 4127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:04.674829 4127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "f2c83d207994493b93f92d1e3e420271"
format_stamp: "Formatted at 2026-09-25 21:56:04 on dist-test-slave-2g0f"
server_key: "1a2090da3e9d4338096d0ca56f8e3c75"
server_key_iv: "39295a11514f0cfd8d335f12e3e4dc83"
server_key_version: "encryptionkey@0"
I20260925 21:56:04.674890 4127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:04.682034 4127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:04.684197 4127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:04.684333 4127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:04.684911 4127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:04.684954 4127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:04.684983 4127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:04.685006 4127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 21:56:04 dist-test-slave-2g0f krb5kdc[4010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:04.691895 4127 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:34879
I20260925 21:56:04.691963 4257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:34879 every 8 connection(s)
I20260925 21:56:04.692301 4127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:04.695170 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:04.685367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:41411 (local address 127.2.70.254:36933)
0925 21:56:04.685418 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:04.685420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:04.685669 (+ 249us) server_negotiation.cc:407] Connection header received
0925 21:56:04.686202 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:04.686203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:04.686218 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:04.686283 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:04.687018 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.687781 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.689369 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.689688 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.689779 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:04.692474 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:04.692475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:04.692476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:04.692485 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:04.693805 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:04.694235 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:04.694236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:04.694236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:04.694298 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:04.694581 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:04.694582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:04.694583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:04.694711 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:04.694776 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:04.695012 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 21:56:04.695100 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 21:56:04.695360 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:04.685374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:36933 (local address 127.2.70.193:41411)
0925 21:56:04.685540 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:04.685586 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:04.686095 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:04.686420 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:04.686425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:04.686713 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:04.686915 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:04.686921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.687931 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.687932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:04.689269 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:04.689273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.689787 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.689788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:04.689918 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:04.690465 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:04.690474 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:04.692346 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:04.693887 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:04.693893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:04.693895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:04.694145 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:04.694358 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:04.694359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:04.694359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:04.694478 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:04.694781 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:04.694786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:04.694912 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 21:56:04.695000 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 21:56:04.695158 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260925 21:56:04.695928 4258 heartbeater.cc:347] Connected to a master server at 127.2.70.254:36933
I20260925 21:56:04.696018 4258 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:04.696194 4258 heartbeater.cc:511] Master 127.2.70.254:36933 requested a full tablet report, sending...
I20260925 21:56:04.696715 4054 ts_manager.cc:194] Registered new tserver with Master: f2c83d207994493b93f92d1e3e420271 (127.2.70.193:34879)
I20260925 21:56:04.697433 4054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:41411
I20260925 21:56:04.700284 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 4127
I20260925 21:56:04.700384 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:04.700564 2331 external_mini_cluster.cc:1428] Setting key 300abaf014b769122347268f45a4165f
I20260925 21:56:04.702203 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:04.696893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:36933 (local address 127.2.70.193:51127)
0925 21:56:04.697022 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:04.697032 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:04.697087 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:04.697208 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:04.697208 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:04.697377 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:04.697428 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:04.697429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.698121 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.698123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:04.698641 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:04.698642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.698963 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.698964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:04.699039 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:04.699459 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:04.699463 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:04.699748 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:04.701366 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:04.701367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:04.701367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:04.701533 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:04.701717 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:04.701718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:04.701718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:04.701763 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:04.701974 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:04.701975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:04.702011 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 21:56:04.702053 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 21:56:04.702136 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260925 21:56:04.702230 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:04.696930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:51127 (local address 127.2.70.254:36933)
0925 21:56:04.696963 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:04.696965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:04.697083 (+ 118us) server_negotiation.cc:407] Connection header received
0925 21:56:04.697140 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:04.697141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:04.697151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:04.697194 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:04.697493 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.698035 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.698711 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.698907 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.698960 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:04.699824 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:04.699826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:04.699826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:04.699902 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:04.701290 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:04.701597 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:04.701598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:04.701598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:04.701658 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:04.701804 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:04.701805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:04.701805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:04.701923 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:04.701985 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:04.702077 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 21:56:04.702148 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:56:04.727269 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:36933
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:04.807605 4267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:04.807868 4267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:04.807907 4267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:04.809105 4267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:04.809146 4267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:04.809166 4267 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 21:56:04.809185 4267 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 21:56:04.809201 4267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:04.810746 4267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.254:36933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:04.810989 4267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:04.811218 4267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:04.816275 4273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:04.816398 4267 server_base.cc:1034] running on GCE node
W20260925 21:56:04.816275 4272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:04.816303 4275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:04.816780 4267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:04.817016 4267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:04.818157 4267 hybrid_clock.cc:648] HybridClock initialized: now 1790373364818137 us; error 31 us; skew 500 ppm
Sep 25 21:56:04 dist-test-slave-2g0f krb5kdc[4010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:04.820751 4267 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:56:04.821206 4267 webserver.cc:460] Webserver started at http://127.2.70.194:38721/ using document root <none> and password file <none>
I20260925 21:56:04.821420 4267 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:04.821470 4267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:04.821593 4267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:04.822610 4267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e"
format_stamp: "Formatted at 2026-09-25 21:56:04 on dist-test-slave-2g0f"
server_key: "349650b8174e233da5bacba09e7fdc98"
server_key_iv: "43902a844b9e3ee70fe3d0d96bee3213"
server_key_version: "encryptionkey@0"
I20260925 21:56:04.822943 4267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e"
format_stamp: "Formatted at 2026-09-25 21:56:04 on dist-test-slave-2g0f"
server_key: "349650b8174e233da5bacba09e7fdc98"
server_key_iv: "43902a844b9e3ee70fe3d0d96bee3213"
server_key_version: "encryptionkey@0"
I20260925 21:56:04.824262 4267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 21:56:04.825045 4282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:04.825205 4267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:04.825292 4267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e"
format_stamp: "Formatted at 2026-09-25 21:56:04 on dist-test-slave-2g0f"
server_key: "349650b8174e233da5bacba09e7fdc98"
server_key_iv: "43902a844b9e3ee70fe3d0d96bee3213"
server_key_version: "encryptionkey@0"
I20260925 21:56:04.825354 4267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:04.842273 4267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:04.844210 4267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:04.844349 4267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:04.844911 4267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:04.844952 4267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:04.844981 4267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:04.844998 4267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 21:56:04 dist-test-slave-2g0f krb5kdc[4010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:04.851608 4267 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:40765
I20260925 21:56:04.851691 4397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:40765 every 8 connection(s)
I20260925 21:56:04.852037 4267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:04.854190 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:04.845335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:58507 (local address 127.2.70.254:36933)
0925 21:56:04.845379 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:04.845381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:04.845589 (+ 208us) server_negotiation.cc:407] Connection header received
0925 21:56:04.846093 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:04.846095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:04.846110 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:04.846190 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:04.847005 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.847761 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.848811 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.849012 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.849068 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:04.851483 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:04.851485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:04.851485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:04.851493 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:04.852955 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:04.853320 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:04.853321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:04.853322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:04.853378 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:04.853584 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:04.853585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:04.853585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:04.853700 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:04.853757 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:04.854006 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 21:56:04.854083 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 21:56:04.854349 4291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:04.845333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:36933 (local address 127.2.70.194:58507)
0925 21:56:04.845476 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:04.845513 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:04.845988 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:04.846268 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:04.846272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:04.846606 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:04.846899 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:04.846906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.847870 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.847871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:04.848725 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:04.848728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.849103 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.849104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:04.849215 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:04.849760 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:04.849771 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:04.851388 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:04.853035 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:04.853041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:04.853043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:04.853255 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:04.853430 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:04.853430 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:04.853430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:04.853531 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:04.853753 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:04.853759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:04.853894 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 21:56:04.853981 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 21:56:04.854138 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260925 21:56:04.854792 4398 heartbeater.cc:347] Connected to a master server at 127.2.70.254:36933
I20260925 21:56:04.854875 4398 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:04.855058 4398 heartbeater.cc:511] Master 127.2.70.254:36933 requested a full tablet report, sending...
I20260925 21:56:04.855413 4055 ts_manager.cc:194] Registered new tserver with Master: a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765)
I20260925 21:56:04.855917 4055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:58507
I20260925 21:56:04.860858 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:04.855877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:43033 (local address 127.2.70.254:36933)
0925 21:56:04.855913 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:04.855915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:04.855967 (+ 52us) server_negotiation.cc:407] Connection header received
0925 21:56:04.856065 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:04.856066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:04.856076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:04.856135 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:04.856448 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.856919 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.857564 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.857785 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.857832 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:04.858753 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:04.858754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:04.858755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:04.858761 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:04.859978 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:04.860259 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:04.860259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:04.860260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:04.860303 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:04.860424 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:04.860425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:04.860425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:04.860548 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:04.860651 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:04.860740 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 21:56:04.860796 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 21:56:04.860867 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:04.855766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:36933 (local address 127.2.70.194:43033)
0925 21:56:04.855910 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:04.855921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:04.855983 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:04.856139 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:04.856140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:04.856308 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:04.856380 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:04.856382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.857008 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.857009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:04.857506 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:04.857508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:04.857860 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:04.857861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:04.857937 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:04.858362 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:04.858366 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:04.858653 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:04.860046 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:04.860046 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:04.860047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:04.860203 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:04.860345 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:04.860345 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:04.860346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:04.860391 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:04.860589 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:04.860590 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:04.860669 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 21:56:04.860717 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 21:56:04.860792 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260925 21:56:04.862012 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 4267
I20260925 21:56:04.862115 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:04.862283 2331 external_mini_cluster.cc:1428] Setting key 1ebc7a923d6409178f90e18ab455f6b2
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:56:04.887282 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:36933
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:04.967939 4407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:04.968156 4407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:04.968190 4407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:04.969367 4407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:04.969411 4407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:04.969439 4407 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 21:56:04.969452 4407 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 21:56:04.969476 4407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:04.970955 4407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.254:36933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:04.971194 4407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:04.971427 4407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:56:04.976605 4407 server_base.cc:1034] running on GCE node
W20260925 21:56:04.976552 4413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:04.976547 4415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:04.976547 4412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:04.977696 4407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:04.977957 4407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:04.979132 4407 hybrid_clock.cc:648] HybridClock initialized: now 1790373364979116 us; error 35 us; skew 500 ppm
Sep 25 21:56:04 dist-test-slave-2g0f krb5kdc[4010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:04.981730 4407 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:56:04.982203 4407 webserver.cc:460] Webserver started at http://127.2.70.195:43991/ using document root <none> and password file <none>
I20260925 21:56:04.982405 4407 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:04.982450 4407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:04.982570 4407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:04.983641 4407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "75e90f4f716443919ee7b3d1db761808"
format_stamp: "Formatted at 2026-09-25 21:56:04 on dist-test-slave-2g0f"
server_key: "3dd759c30155303d0429b844bd6b18a1"
server_key_iv: "1568366755811451974b221a20449492"
server_key_version: "encryptionkey@0"
I20260925 21:56:04.983994 4407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "75e90f4f716443919ee7b3d1db761808"
format_stamp: "Formatted at 2026-09-25 21:56:04 on dist-test-slave-2g0f"
server_key: "3dd759c30155303d0429b844bd6b18a1"
server_key_iv: "1568366755811451974b221a20449492"
server_key_version: "encryptionkey@0"
I20260925 21:56:04.985255 4407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:04.985980 4422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:04.986176 4407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:04.986255 4407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "75e90f4f716443919ee7b3d1db761808"
format_stamp: "Formatted at 2026-09-25 21:56:04 on dist-test-slave-2g0f"
server_key: "3dd759c30155303d0429b844bd6b18a1"
server_key_iv: "1568366755811451974b221a20449492"
server_key_version: "encryptionkey@0"
I20260925 21:56:04.986320 4407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:05.007582 4407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:05.009909 4407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:05.010043 4407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:05.010635 4407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:05.010679 4407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:05.010710 4407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:05.010725 4407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:05.017457 4407 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:46663
I20260925 21:56:05.017585 4537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:46663 every 8 connection(s)
I20260925 21:56:05.017848 4407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
Sep 25 21:56:05 dist-test-slave-2g0f krb5kdc[4010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:05.021322 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.011130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:59659 (local address 127.2.70.254:36933)
0925 21:56:05.011183 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.011185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.011419 (+ 234us) server_negotiation.cc:407] Connection header received
0925 21:56:05.011987 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.011988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.012001 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.012038 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:05.013034 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.013817 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.015379 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.015672 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.015723 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.018418 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:05.018420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:05.018420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:05.018429 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:05.019776 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:05.020291 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:05.020292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:05.020292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:05.020367 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:05.020665 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:05.020667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:05.020667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:05.020806 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:05.020877 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.021176 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.021254 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 21:56:05.021430 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.011120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:36933 (local address 127.2.70.195:59659)
0925 21:56:05.011282 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.011333 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.011875 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.012228 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.012232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.012607 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:05.012909 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.012917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.013921 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.013922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.015272 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.015275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.015756 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.015758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.015944 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.016585 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:05.016596 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:05.018317 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:05.019962 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:05.019968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:05.019970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:05.020197 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:05.020461 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:05.020462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:05.020462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:05.020578 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:05.020880 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:05.020884 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:05.021012 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.021101 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.021242 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260925 21:56:05.022140 4538 heartbeater.cc:347] Connected to a master server at 127.2.70.254:36933
I20260925 21:56:05.022240 4538 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:05.022336 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 4407
I20260925 21:56:05.022426 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:05.022439 4538 heartbeater.cc:511] Master 127.2.70.254:36933 requested a full tablet report, sending...
I20260925 21:56:05.022622 2331 external_mini_cluster.cc:1428] Setting key 17fd73e92b7f1a172e03926e9741328b
I20260925 21:56:05.022905 4055 ts_manager.cc:194] Registered new tserver with Master: 75e90f4f716443919ee7b3d1db761808 (127.2.70.195:46663)
I20260925 21:56:05.023383 4055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:59659
I20260925 21:56:05.023764 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:56:05.029597 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.023351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:36933 (local address 127.2.70.195:46373)
0925 21:56:05.023650 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.023661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.023726 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.023893 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.023894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.024098 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:05.024161 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.024163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.025017 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.025018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.025627 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.025630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.025963 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.025965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.026034 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.026452 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:05.026457 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:05.026875 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:05.028686 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:05.028687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:05.028688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:05.028853 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:05.029040 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:05.029041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:05.029041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:05.029089 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:05.029316 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:05.029317 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:05.029357 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.029407 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.029490 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":74,"threads_started":1}
I20260925 21:56:05.029616 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.023404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:46373 (local address 127.2.70.254:36933)
0925 21:56:05.023442 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.023444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.023714 (+ 270us) server_negotiation.cc:407] Connection header received
0925 21:56:05.023777 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.023778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.023787 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.023849 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:05.024246 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.024917 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.025721 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.025908 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.025951 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.026973 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:05.026975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:05.026975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:05.026982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:05.028590 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:05.028919 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:05.028920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:05.028920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:05.028983 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:05.029135 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:05.029135 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:05.029136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:05.029260 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:05.029322 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.029443 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.029517 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 25 21:56:05 dist-test-slave-2g0f krb5kdc[4010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 21:56:05 dist-test-slave-2g0f krb5kdc[4010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:05.046546 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.039573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48648 (local address 127.2.70.254:36933)
0925 21:56:05.039644 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.039646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.039656 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:05.039682 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.039683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.039692 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.039747 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:05.040336 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.040836 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.041652 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.041859 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.041922 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.044142 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:05.044144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:05.044144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:05.044152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:05.045463 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:05.045814 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:05.045815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:05.045816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:05.045872 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:05.046109 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:05.046110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:05.046110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:05.046239 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:05.046330 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.046407 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.046476 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 21:56:05.048418 4054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:05.050006 4054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:05.056209 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.052282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:34879 (local address 127.0.0.1:41582)
0925 21:56:05.052459 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.052499 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.052563 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.052841 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.052844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.052846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:05.052916 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.052918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.053883 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.053884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.054923 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.054926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.055911 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.055912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.055989 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.056003 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.056074 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.056125 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
I20260925 21:56:05.056232 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.052473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.2.70.193:34879)
0925 21:56:05.052649 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.052653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.052665 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:56:05.052734 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.052736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.052748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.052821 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:05.052981 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.053810 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.055040 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.055793 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.055910 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.055981 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.056089 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.056128 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
I20260925 21:56:05.056641 4555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.052283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:40765 (local address 127.0.0.1:47434)
0925 21:56:05.052699 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.052708 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.052751 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.052995 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.052996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.052998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:05.053049 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.053050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.054423 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.054424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.055415 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.055417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.056411 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.056412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.056474 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.056478 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.056522 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.056559 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":4,"thread_start_us":83,"threads_started":1}
I20260925 21:56:05.056890 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.052540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.2.70.194:40765)
0925 21:56:05.052878 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.052881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.052889 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:05.052917 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.052919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.052929 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.052987 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:05.053206 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.054324 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.055484 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.056247 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.056313 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.056376 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.056756 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.056801 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":77,"threads_started":1}
I20260925 21:56:05.056911 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.052283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:46663 (local address 127.0.0.1:35124)
0925 21:56:05.052749 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.052757 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.052790 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.053258 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.053259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.053261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:05.053293 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.053294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.054520 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.054521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.055627 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.055630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.056694 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.056695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.056763 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.056767 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.056810 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.056848 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":22,"thread_start_us":79,"threads_started":1}
I20260925 21:56:05.057010 4192 tablet_service.cc:1467] Processing CreateTablet for tablet 07aeb10079d6484abacf4cdbc99632e8 (DEFAULT_TABLE table=test-table [id=6b1ad60087494393891e7db414dfce0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:05.057394 4192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07aeb10079d6484abacf4cdbc99632e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:05.057564 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.052518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.2.70.195:46663)
0925 21:56:05.053107 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.053111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.053119 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:05.053153 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.053155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.053170 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.053354 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:05.053410 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.054450 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.055710 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.056621 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.056917 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.057358 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.057384 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.057424 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":65,"threads_started":1}
I20260925 21:56:05.057578 4332 tablet_service.cc:1467] Processing CreateTablet for tablet 07aeb10079d6484abacf4cdbc99632e8 (DEFAULT_TABLE table=test-table [id=6b1ad60087494393891e7db414dfce0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:05.057899 4332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07aeb10079d6484abacf4cdbc99632e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:05.058100 4472 tablet_service.cc:1467] Processing CreateTablet for tablet 07aeb10079d6484abacf4cdbc99632e8 (DEFAULT_TABLE table=test-table [id=6b1ad60087494393891e7db414dfce0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:05.058363 4472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07aeb10079d6484abacf4cdbc99632e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:05.060803 4561 tablet_bootstrap.cc:492] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808: Bootstrap starting.
I20260925 21:56:05.061170 4562 tablet_bootstrap.cc:492] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271: Bootstrap starting.
I20260925 21:56:05.061821 4561 tablet_bootstrap.cc:654] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:05.061821 4562 tablet_bootstrap.cc:654] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:05.062148 4562 log.cc:824] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:05.062148 4561 log.cc:824] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:05.062474 4563 tablet_bootstrap.cc:492] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e: Bootstrap starting.
I20260925 21:56:05.062918 4562 tablet_bootstrap.cc:492] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271: No bootstrap required, opened a new log
I20260925 21:56:05.062990 4562 ts_tablet_manager.cc:1397] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:05.063035 4561 tablet_bootstrap.cc:492] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808: No bootstrap required, opened a new log
I20260925 21:56:05.063104 4561 ts_tablet_manager.cc:1397] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:56:05.063164 4563 tablet_bootstrap.cc:654] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:05.063545 4563 log.cc:824] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:05.064297 4563 tablet_bootstrap.cc:492] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e: No bootstrap required, opened a new log
I20260925 21:56:05.064369 4563 ts_tablet_manager.cc:1397] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:05.064800 4561 raft_consensus.cc:348] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } }
I20260925 21:56:05.064937 4561 raft_consensus.cc:374] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:05.064970 4561 raft_consensus.cc:729] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e90f4f716443919ee7b3d1db761808, State: Initialized, Role: FOLLOWER
I20260925 21:56:05.065022 4562 raft_consensus.cc:348] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } }
I20260925 21:56:05.065116 4561 consensus_queue.cc:260] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } }
I20260925 21:56:05.065188 4562 raft_consensus.cc:374] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:05.065230 4562 raft_consensus.cc:729] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c83d207994493b93f92d1e3e420271, State: Initialized, Role: FOLLOWER
I20260925 21:56:05.065341 4562 consensus_queue.cc:260] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } }
I20260925 21:56:05.065412 4561 ts_tablet_manager.cc:1428] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 21:56:05.065542 4538 heartbeater.cc:503] Master 127.2.70.254:36933 was elected leader, sending a full tablet report...
I20260925 21:56:05.065604 4562 ts_tablet_manager.cc:1428] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:56:05.065723 4258 heartbeater.cc:503] Master 127.2.70.254:36933 was elected leader, sending a full tablet report...
I20260925 21:56:05.066280 4563 raft_consensus.cc:348] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } }
I20260925 21:56:05.066430 4563 raft_consensus.cc:374] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:05.066462 4563 raft_consensus.cc:729] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a845e9efd2cc4525ab38d8f81dd0ef1e, State: Initialized, Role: FOLLOWER
I20260925 21:56:05.066557 4563 consensus_queue.cc:260] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } }
I20260925 21:56:05.066815 4563 ts_tablet_manager.cc:1428] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 21:56:05.066938 4398 heartbeater.cc:503] Master 127.2.70.254:36933 was elected leader, sending a full tablet report...
W20260925 21:56:05.102631 4399 tablet.cc:2369] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:05.164053 4568 raft_consensus.cc:482] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:05.164209 4568 raft_consensus.cc:504] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } }
I20260925 21:56:05.164647 4568 leader_election.cc:288] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75e90f4f716443919ee7b3d1db761808 (127.2.70.195:46663), a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765)
I20260925 21:56:05.167591 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.164920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:49457 (local address 127.2.70.195:46663)
0925 21:56:05.164986 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.164988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.164999 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:56:05.165025 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.165026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.165046 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.165109 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:05.165293 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.165900 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.166905 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.167265 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.167324 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.167334 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.167491 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.167526 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 21:56:05.167635 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.164841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:40765 (local address 127.2.70.193:41079)
0925 21:56:05.165192 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.165202 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.165264 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.165379 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.165380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.165381 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:05.165456 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.165457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.166041 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.166042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.166821 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.166823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.167357 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.167358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.167418 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.167421 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.167465 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.167505 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":185,"thread_start_us":117,"threads_started":1}
I20260925 21:56:05.167887 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.164935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:41079 (local address 127.2.70.194:40765)
0925 21:56:05.164987 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.164989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.165263 (+ 274us) server_negotiation.cc:407] Connection header received
0925 21:56:05.165318 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.165319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.165330 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.165376 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:05.165513 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.165962 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.166906 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.167270 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.167342 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.167351 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.167741 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.167791 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 21:56:05.167637 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.164840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:46663 (local address 127.2.70.193:49457)
0925 21:56:05.164892 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.164904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.164951 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.165130 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.165132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.165134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:05.165208 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.165210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.165988 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.165990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.166808 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.166811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.167344 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.167344 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.167422 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.167425 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.167481 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.167527 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1}
I20260925 21:56:05.168218 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07aeb10079d6484abacf4cdbc99632e8" candidate_uuid: "f2c83d207994493b93f92d1e3e420271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" is_pre_election: true
I20260925 21:56:05.168401 4352 raft_consensus.cc:2454] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2c83d207994493b93f92d1e3e420271 in term 0.
I20260925 21:56:05.168497 4492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07aeb10079d6484abacf4cdbc99632e8" candidate_uuid: "f2c83d207994493b93f92d1e3e420271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e90f4f716443919ee7b3d1db761808" is_pre_election: true
I20260925 21:56:05.168628 4143 leader_election.cc:302] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a845e9efd2cc4525ab38d8f81dd0ef1e, f2c83d207994493b93f92d1e3e420271; no voters:
I20260925 21:56:05.168656 4492 raft_consensus.cc:2454] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2c83d207994493b93f92d1e3e420271 in term 0.
I20260925 21:56:05.168749 4568 raft_consensus.cc:2781] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:05.168805 4568 raft_consensus.cc:482] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:05.168824 4568 raft_consensus.cc:3037] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:05.169565 4568 raft_consensus.cc:504] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } }
I20260925 21:56:05.169760 4568 leader_election.cc:288] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [CANDIDATE]: Term 1 election: Requested vote from peers 75e90f4f716443919ee7b3d1db761808 (127.2.70.195:46663), a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765)
I20260925 21:56:05.169854 4492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07aeb10079d6484abacf4cdbc99632e8" candidate_uuid: "f2c83d207994493b93f92d1e3e420271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e90f4f716443919ee7b3d1db761808"
I20260925 21:56:05.169849 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07aeb10079d6484abacf4cdbc99632e8" candidate_uuid: "f2c83d207994493b93f92d1e3e420271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e"
I20260925 21:56:05.169931 4492 raft_consensus.cc:3037] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:05.169943 4352 raft_consensus.cc:3037] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:05.170568 4492 raft_consensus.cc:2454] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2c83d207994493b93f92d1e3e420271 in term 1.
I20260925 21:56:05.170730 4352 raft_consensus.cc:2454] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2c83d207994493b93f92d1e3e420271 in term 1.
I20260925 21:56:05.171070 4146 leader_election.cc:302] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e90f4f716443919ee7b3d1db761808, f2c83d207994493b93f92d1e3e420271; no voters:
I20260925 21:56:05.171159 4568 raft_consensus.cc:2781] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:05.171331 4568 raft_consensus.cc:686] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 1 LEADER]: Becoming Leader. State: Replica: f2c83d207994493b93f92d1e3e420271, State: Running, Role: LEADER
I20260925 21:56:05.171425 4568 consensus_queue.cc:237] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } }
I20260925 21:56:05.172456 4055 catalog_manager.cc:5495] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2c83d207994493b93f92d1e3e420271 (127.2.70.193). New cstate: current_term: 1 leader_uuid: "f2c83d207994493b93f92d1e3e420271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } health_report { overall_health: UNKNOWN } } }
W20260925 21:56:05.193161 4259 tablet.cc:2369] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 21:56:05 dist-test-slave-2g0f krb5kdc[4010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:05.224368 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.216608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:36933 (local address 127.0.0.1:48650)
0925 21:56:05.216808 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.216819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.216916 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.217166 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.217167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.217618 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:05.217680 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.217683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.218467 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.218468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.219158 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.219160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.219609 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.219610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.219681 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.220474 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:05.220479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:05.221930 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:05.223289 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:05.223299 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:05.223301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:05.223527 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:05.223738 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:05.223739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:05.223739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:05.223799 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:05.224035 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:05.224041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:05.224086 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.224139 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.224235 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":61,"threads_started":1}
I20260925 21:56:05.224367 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.216767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.2.70.254:36933)
0925 21:56:05.216819 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.216821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.216886 (+ 65us) server_negotiation.cc:407] Connection header received
0925 21:56:05.217016 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.217017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.217048 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.217105 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:05.217766 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.218367 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.219237 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.219512 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.219607 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.222021 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:05.222022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:05.222023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:05.222030 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:05.223206 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:05.223589 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:05.223590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:05.223590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:05.223642 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:05.223865 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:05.223866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:05.223866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:05.223981 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:05.224030 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.224175 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.224281 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":22}
I20260925 21:56:05.226735 4055 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:48650:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 21:56:05.226943 4055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:05.228754 4472 tablet_service.cc:1467] Processing CreateTablet for tablet aaeeea742639426ea7ace69748e71b58 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3d1bf3a00f74180aab87d824c574436]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 21:56:05.228750 4192 tablet_service.cc:1467] Processing CreateTablet for tablet aaeeea742639426ea7ace69748e71b58 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3d1bf3a00f74180aab87d824c574436]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 21:56:05.228760 4332 tablet_service.cc:1467] Processing CreateTablet for tablet aaeeea742639426ea7ace69748e71b58 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3d1bf3a00f74180aab87d824c574436]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 21:56:05.228927 4472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeeea742639426ea7ace69748e71b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:05.228927 4192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeeea742639426ea7ace69748e71b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:05.228943 4332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeeea742639426ea7ace69748e71b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:05.230203 4561 tablet_bootstrap.cc:492] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: Bootstrap starting.
I20260925 21:56:05.230361 4562 tablet_bootstrap.cc:492] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271: Bootstrap starting.
I20260925 21:56:05.230388 4563 tablet_bootstrap.cc:492] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e: Bootstrap starting.
I20260925 21:56:05.230756 4562 tablet_bootstrap.cc:654] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:05.230772 4561 tablet_bootstrap.cc:654] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:05.230969 4563 tablet_bootstrap.cc:654] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:05.231559 4562 tablet_bootstrap.cc:492] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271: No bootstrap required, opened a new log
I20260925 21:56:05.231604 4562 ts_tablet_manager.cc:1397] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:05.231786 4562 raft_consensus.cc:348] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.231909 4562 raft_consensus.cc:374] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:05.231941 4563 tablet_bootstrap.cc:492] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e: No bootstrap required, opened a new log
I20260925 21:56:05.231957 4562 raft_consensus.cc:729] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c83d207994493b93f92d1e3e420271, State: Initialized, Role: FOLLOWER
I20260925 21:56:05.231989 4563 ts_tablet_manager.cc:1397] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:05.232002 4562 consensus_queue.cc:260] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.232051 4561 tablet_bootstrap.cc:492] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: No bootstrap required, opened a new log
I20260925 21:56:05.232088 4562 ts_tablet_manager.cc:1428] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:05.232101 4561 ts_tablet_manager.cc:1397] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:05.232136 4563 raft_consensus.cc:348] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.232213 4563 raft_consensus.cc:374] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:05.232232 4563 raft_consensus.cc:729] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a845e9efd2cc4525ab38d8f81dd0ef1e, State: Initialized, Role: FOLLOWER
I20260925 21:56:05.232223 4568 tablet_replica.cc:439] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } }
I20260925 21:56:05.232281 4568 tablet_replica.cc:442] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:05.232259 4561 raft_consensus.cc:348] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.232280 4563 consensus_queue.cc:260] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.232316 4561 raft_consensus.cc:374] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:05.232337 4561 raft_consensus.cc:729] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e90f4f716443919ee7b3d1db761808, State: Initialized, Role: FOLLOWER
I20260925 21:56:05.232359 4563 ts_tablet_manager.cc:1428] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:05.232376 4561 consensus_queue.cc:260] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.232455 4561 ts_tablet_manager.cc:1428] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:05.232602 4569 tablet_replica.cc:439] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } }
I20260925 21:56:05.232697 4569 tablet_replica.cc:442] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:05.232846 4567 tablet_replica.cc:439] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } }
I20260925 21:56:05.232939 4567 tablet_replica.cc:442] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 21:56:05.268415 4539 tablet.cc:2369] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:05.300905 4567 raft_consensus.cc:482] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:05.301010 4567 raft_consensus.cc:504] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.301348 4567 leader_election.cc:288] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2c83d207994493b93f92d1e3e420271 (127.2.70.193:34879), a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765)
I20260925 21:56:05.304075 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.301541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:34879 (local address 127.2.70.195:55763)
0925 21:56:05.301595 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.301609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.301673 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.301855 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.301855 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.301857 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:05.301920 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.301922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.302527 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.302528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.303288 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.303290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.303806 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.303807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.303902 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.303905 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.303952 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.303994 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 21:56:05.304068 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.301675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:55763 (local address 127.2.70.193:34879)
0925 21:56:05.301722 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.301724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.301734 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:05.301766 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.301767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.301779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.301834 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:05.301980 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.302451 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.303378 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.303738 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.303799 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.303808 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.303963 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.303999 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 21:56:05.304085 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.301675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:58079 (local address 127.2.70.194:40765)
0925 21:56:05.301725 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.301727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.301863 (+ 136us) server_negotiation.cc:407] Connection header received
0925 21:56:05.301898 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.301899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.301908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.301957 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:05.302088 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.302545 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.303414 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.303754 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.303814 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.303823 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.303983 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.304019 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 21:56:05.304100 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.301549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:40765 (local address 127.2.70.195:58079)
0925 21:56:05.301802 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.301810 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.301854 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.301966 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.301967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.301969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:05.302035 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.302036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.302616 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.302617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.303338 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.303340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.303860 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.303861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.303922 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.303924 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.303969 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.304007 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":49,"thread_start_us":111,"threads_started":1}
I20260925 21:56:05.304432 4212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeeea742639426ea7ace69748e71b58" candidate_uuid: "75e90f4f716443919ee7b3d1db761808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c83d207994493b93f92d1e3e420271" is_pre_election: true
I20260925 21:56:05.304540 4212 raft_consensus.cc:2454] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e90f4f716443919ee7b3d1db761808 in term 0.
I20260925 21:56:05.304602 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeeea742639426ea7ace69748e71b58" candidate_uuid: "75e90f4f716443919ee7b3d1db761808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" is_pre_election: true
I20260925 21:56:05.304688 4352 raft_consensus.cc:2454] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e90f4f716443919ee7b3d1db761808 in term 0.
I20260925 21:56:05.304853 4423 leader_election.cc:302] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e90f4f716443919ee7b3d1db761808, a845e9efd2cc4525ab38d8f81dd0ef1e; no voters:
I20260925 21:56:05.304986 4567 raft_consensus.cc:2781] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:05.305027 4567 raft_consensus.cc:482] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:05.305047 4567 raft_consensus.cc:3037] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:05.305706 4567 raft_consensus.cc:504] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.305836 4567 leader_election.cc:288] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [CANDIDATE]: Term 1 election: Requested vote from peers f2c83d207994493b93f92d1e3e420271 (127.2.70.193:34879), a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765)
I20260925 21:56:05.306031 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeeea742639426ea7ace69748e71b58" candidate_uuid: "75e90f4f716443919ee7b3d1db761808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e"
I20260925 21:56:05.306051 4212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeeea742639426ea7ace69748e71b58" candidate_uuid: "75e90f4f716443919ee7b3d1db761808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c83d207994493b93f92d1e3e420271"
I20260925 21:56:05.306111 4352 raft_consensus.cc:3037] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:05.306116 4212 raft_consensus.cc:3037] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:05.306852 4212 raft_consensus.cc:2454] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e90f4f716443919ee7b3d1db761808 in term 1.
I20260925 21:56:05.306852 4352 raft_consensus.cc:2454] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e90f4f716443919ee7b3d1db761808 in term 1.
I20260925 21:56:05.307054 4425 leader_election.cc:302] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e90f4f716443919ee7b3d1db761808, f2c83d207994493b93f92d1e3e420271; no voters:
I20260925 21:56:05.307154 4567 raft_consensus.cc:2781] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:05.307348 4567 raft_consensus.cc:686] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [term 1 LEADER]: Becoming Leader. State: Replica: 75e90f4f716443919ee7b3d1db761808, State: Running, Role: LEADER
I20260925 21:56:05.307466 4567 consensus_queue.cc:237] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.308032 4583 tablet_replica.cc:439] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: TxnStatusTablet state changed. Reason: New leader 75e90f4f716443919ee7b3d1db761808. Latest consensus state: current_term: 1 leader_uuid: "75e90f4f716443919ee7b3d1db761808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } }
I20260925 21:56:05.308135 4583 tablet_replica.cc:442] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: This TxnStatusTablet replica's current role is: LEADER
I20260925 21:56:05.308314 4585 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 21:56:05.308431 4585 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 21:56:05.308346 4052 catalog_manager.cc:5495] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75e90f4f716443919ee7b3d1db761808 (127.2.70.195). New cstate: current_term: 1 leader_uuid: "75e90f4f716443919ee7b3d1db761808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } health_report { overall_health: HEALTHY } } }
I20260925 21:56:05.330544 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.327522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35130 (local address 127.2.70.195:46663)
0925 21:56:05.327584 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.327586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.327594 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:05.327627 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.327627 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.327704 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.327774 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:05.327970 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.328528 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.329688 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.329889 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.329950 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.330214 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:05.330320 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:05.330387 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.330440 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.330461 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 21:56:05.330580 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.327247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:46663 (local address 127.0.0.1:35130)
0925 21:56:05.327332 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.327349 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.327418 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.327805 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.327806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.327808 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 21:56:05.327898 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.327900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.328607 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.328609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.329586 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.329589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.329961 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.329962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.330034 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.330054 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 21:56:05.330383 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 21:56:05.330390 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.330439 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.330493 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 21:56:05.332542 4352 raft_consensus.cc:1261] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Refusing update from remote peer 75e90f4f716443919ee7b3d1db761808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:05.332542 4212 raft_consensus.cc:1261] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Refusing update from remote peer 75e90f4f716443919ee7b3d1db761808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:05.332857 4567 consensus_queue.cc:1035] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:05.333067 4567 consensus_queue.cc:1035] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:05.334091 4569 tablet_replica.cc:439] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e: TxnStatusTablet state changed. Reason: New leader 75e90f4f716443919ee7b3d1db761808. Latest consensus state: current_term: 1 leader_uuid: "75e90f4f716443919ee7b3d1db761808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } }
I20260925 21:56:05.334175 4569 tablet_replica.cc:442] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:05.334179 4568 tablet_replica.cc:439] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271: TxnStatusTablet state changed. Reason: New leader 75e90f4f716443919ee7b3d1db761808. Latest consensus state: current_term: 1 leader_uuid: "75e90f4f716443919ee7b3d1db761808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } }
I20260925 21:56:05.334278 4568 tablet_replica.cc:442] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:05.335160 4583 tablet_replica.cc:439] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75e90f4f716443919ee7b3d1db761808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } }
I20260925 21:56:05.335891 4568 tablet_replica.cc:439] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75e90f4f716443919ee7b3d1db761808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } }
I20260925 21:56:05.335975 4568 tablet_replica.cc:442] T aaeeea742639426ea7ace69748e71b58 P f2c83d207994493b93f92d1e3e420271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:05.336293 4583 tablet_replica.cc:442] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: This TxnStatusTablet replica's current role is: LEADER
I20260925 21:56:05.336424 4590 mvcc.cc:205] Tried to move back new op lower bound from 7333369304399577088 to 7333369304300023808. Current Snapshot: MvccSnapshot[applied={T|T < 7333369304399577088 or (T in {7333369304399577088})}]
I20260925 21:56:05.340341 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.337379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.2.70.193:34879)
0925 21:56:05.337419 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.337421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.337454 (+ 33us) server_negotiation.cc:407] Connection header received
0925 21:56:05.337482 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.337483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.337498 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.337547 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:05.337674 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.338164 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.339340 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.339604 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.339662 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.339809 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:05.339953 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:05.340035 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.340250 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.340274 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 21:56:05.340889 4567 tablet_replica.cc:439] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75e90f4f716443919ee7b3d1db761808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } }
I20260925 21:56:05.340976 4567 tablet_replica.cc:442] T aaeeea742639426ea7ace69748e71b58 P 75e90f4f716443919ee7b3d1db761808: This TxnStatusTablet replica's current role is: LEADER
I20260925 21:56:05.341189 4569 tablet_replica.cc:439] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75e90f4f716443919ee7b3d1db761808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } } }
I20260925 21:56:05.341280 4569 tablet_replica.cc:442] T aaeeea742639426ea7ace69748e71b58 P a845e9efd2cc4525ab38d8f81dd0ef1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:05.345590 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.343203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:58943 (local address 127.2.70.195:46663)
0925 21:56:05.343243 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.343245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.343372 (+ 127us) server_negotiation.cc:407] Connection header received
0925 21:56:05.343428 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.343429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.343440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.343479 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:05.343608 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.344178 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.344924 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.345125 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.345180 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.345355 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:05.345385 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:05.345426 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.345486 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.345505 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 21:56:05.345610 4600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.343141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:46663 (local address 127.2.70.193:58943)
0925 21:56:05.343312 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.343321 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.343375 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.343490 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.343491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.343492 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 21:56:05.343558 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.343559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.344250 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.344251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.344853 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.344855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.345198 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.345199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.345264 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.345278 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 21:56:05.345437 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 21:56:05.345439 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.345474 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.345517 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260925 21:56:05.347913 4352 raft_consensus.cc:1261] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Refusing update from remote peer f2c83d207994493b93f92d1e3e420271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:05.348042 4492 raft_consensus.cc:1261] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Refusing update from remote peer f2c83d207994493b93f92d1e3e420271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:05.348160 4571 consensus_queue.cc:1035] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:05.348284 4568 consensus_queue.cc:1035] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:05.351080 4588 mvcc.cc:205] Tried to move back new op lower bound from 7333369304463515648 to 7333369303742836736. Current Snapshot: MvccSnapshot[applied={T|T < 7333369304463515648}]
I20260925 21:56:05.356658 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.354142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:49869 (local address 127.2.70.193:34879)
0925 21:56:05.354176 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:05.354178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:05.354227 (+ 49us) server_negotiation.cc:407] Connection header received
0925 21:56:05.354295 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:05.354296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:05.354305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:05.354365 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:05.354528 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.355031 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.355941 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.356233 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.356285 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.356434 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:05.356469 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:05.356512 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:05.356562 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 21:56:05.356583 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 21:56:05.356673 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:05.354104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:34879 (local address 127.2.70.195:49869)
0925 21:56:05.354184 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:05.354194 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:05.354241 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:05.354398 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:05.354399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:05.354401 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 21:56:05.354471 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.354473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.355103 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.355104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.355873 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:05.355876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:05.356288 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:05.356289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:05.356357 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:05.356370 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 21:56:05.356521 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 21:56:05.356523 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:05.356561 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 21:56:05.356601 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 21:56:05.363472 4053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:05.363613 4053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:05.364387 4053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337: Sending DeleteTablet for 3 replicas of tablet 07aeb10079d6484abacf4cdbc99632e8
I20260925 21:56:05.364732 4472 tablet_service.cc:1514] Processing DeleteTablet for tablet 07aeb10079d6484abacf4cdbc99632e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:05 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:35124
I20260925 21:56:05.364775 4192 tablet_service.cc:1514] Processing DeleteTablet for tablet 07aeb10079d6484abacf4cdbc99632e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:05 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:41582
I20260925 21:56:05.364966 4608 tablet_replica.cc:330] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271: stopping tablet replica
I20260925 21:56:05.365002 4332 tablet_service.cc:1514] Processing DeleteTablet for tablet 07aeb10079d6484abacf4cdbc99632e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:05 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:47434
I20260925 21:56:05.365074 4608 raft_consensus.cc:2229] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:05.365201 4609 tablet_replica.cc:330] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e: stopping tablet replica
I20260925 21:56:05.365202 4608 raft_consensus.cc:2258] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:05.365360 4609 raft_consensus.cc:2229] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:05.365465 4609 raft_consensus.cc:2258] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:05.365465 4053 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:05.365625 4053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:05.365844 4608 ts_tablet_manager.cc:1905] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:05.365880 4609 ts_tablet_manager.cc:1905] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:05.367146 4192 tablet_service.cc:1467] Processing CreateTablet for tablet 53c6e24abd6643e0a69c6ace3f214c33 (DEFAULT_TABLE table=test-table [id=98d6ae3d8a2749c8afc00b787b6ba386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:05.367254 4192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c6e24abd6643e0a69c6ace3f214c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:05.367718 4609 ts_tablet_manager.cc:1918] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:05.367805 4609 log.cc:1196] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/07aeb10079d6484abacf4cdbc99632e8
I20260925 21:56:05.367815 4608 ts_tablet_manager.cc:1918] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:05.367908 4608 log.cc:1196] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/wal/wals/07aeb10079d6484abacf4cdbc99632e8
I20260925 21:56:05.368072 4609 ts_tablet_manager.cc:1939] T 07aeb10079d6484abacf4cdbc99632e8 P a845e9efd2cc4525ab38d8f81dd0ef1e: Deleting consensus metadata
I20260925 21:56:05.368114 4608 ts_tablet_manager.cc:1939] T 07aeb10079d6484abacf4cdbc99632e8 P f2c83d207994493b93f92d1e3e420271: Deleting consensus metadata
I20260925 21:56:05.368449 4562 tablet_bootstrap.cc:492] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271: Bootstrap starting.
I20260925 21:56:05.368494 4041 catalog_manager.cc:4841] TS f2c83d207994493b93f92d1e3e420271 (127.2.70.193:34879): tablet 07aeb10079d6484abacf4cdbc99632e8 (table test-table [id=6b1ad60087494393891e7db414dfce0b]) successfully deleted
I20260925 21:56:05.368669 4472 tablet_service.cc:1467] Processing CreateTablet for tablet 53c6e24abd6643e0a69c6ace3f214c33 (DEFAULT_TABLE table=test-table [id=98d6ae3d8a2749c8afc00b787b6ba386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:05.368779 4472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c6e24abd6643e0a69c6ace3f214c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:05.368808 4562 tablet_bootstrap.cc:654] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:05.369306 4562 tablet_bootstrap.cc:492] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271: No bootstrap required, opened a new log
I20260925 21:56:05.369338 4562 ts_tablet_manager.cc:1397] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:05.369433 4562 raft_consensus.cc:348] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.369479 4562 raft_consensus.cc:374] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:05.369490 4562 raft_consensus.cc:729] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c83d207994493b93f92d1e3e420271, State: Initialized, Role: FOLLOWER
I20260925 21:56:05.369529 4562 consensus_queue.cc:260] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.369606 4562 ts_tablet_manager.cc:1428] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:05.370096 4561 tablet_bootstrap.cc:492] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808: Bootstrap starting.
I20260925 21:56:05.370443 4561 tablet_bootstrap.cc:654] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:05.371227 4561 tablet_bootstrap.cc:492] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808: No bootstrap required, opened a new log
I20260925 21:56:05.371273 4561 ts_tablet_manager.cc:1397] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:05.371412 4561 raft_consensus.cc:348] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.371470 4561 raft_consensus.cc:374] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:05.371488 4561 raft_consensus.cc:729] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e90f4f716443919ee7b3d1db761808, State: Initialized, Role: FOLLOWER
I20260925 21:56:05.371483 4039 catalog_manager.cc:4841] TS a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765): tablet 07aeb10079d6484abacf4cdbc99632e8 (table test-table [id=6b1ad60087494393891e7db414dfce0b]) successfully deleted
I20260925 21:56:05.371526 4561 consensus_queue.cc:260] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.371606 4561 ts_tablet_manager.cc:1428] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:05.371609 4332 tablet_service.cc:1467] Processing CreateTablet for tablet 53c6e24abd6643e0a69c6ace3f214c33 (DEFAULT_TABLE table=test-table [id=98d6ae3d8a2749c8afc00b787b6ba386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:05.371702 4332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c6e24abd6643e0a69c6ace3f214c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:05.371914 4607 tablet_replica.cc:330] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808: stopping tablet replica
I20260925 21:56:05.372089 4607 raft_consensus.cc:2229] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:05.372148 4607 raft_consensus.cc:2258] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:05.372898 4563 tablet_bootstrap.cc:492] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e: Bootstrap starting.
I20260925 21:56:05.373279 4563 tablet_bootstrap.cc:654] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:05.373279 4607 ts_tablet_manager.cc:1905] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:05.374862 4607 ts_tablet_manager.cc:1918] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:05.374926 4607 log.cc:1196] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/07aeb10079d6484abacf4cdbc99632e8
I20260925 21:56:05.375181 4607 ts_tablet_manager.cc:1939] T 07aeb10079d6484abacf4cdbc99632e8 P 75e90f4f716443919ee7b3d1db761808: Deleting consensus metadata
I20260925 21:56:05.375623 4042 catalog_manager.cc:4841] TS 75e90f4f716443919ee7b3d1db761808 (127.2.70.195:46663): tablet 07aeb10079d6484abacf4cdbc99632e8 (table test-table [id=6b1ad60087494393891e7db414dfce0b]) successfully deleted
I20260925 21:56:05.376192 4563 tablet_bootstrap.cc:492] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e: No bootstrap required, opened a new log
I20260925 21:56:05.376238 4563 ts_tablet_manager.cc:1397] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 21:56:05.376384 4563 raft_consensus.cc:348] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.376436 4563 raft_consensus.cc:374] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:05.376447 4563 raft_consensus.cc:729] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a845e9efd2cc4525ab38d8f81dd0ef1e, State: Initialized, Role: FOLLOWER
I20260925 21:56:05.376482 4563 consensus_queue.cc:260] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.376559 4563 ts_tablet_manager.cc:1428] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:05.383651 4603 raft_consensus.cc:482] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:05.383739 4603 raft_consensus.cc:504] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.384106 4492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c6e24abd6643e0a69c6ace3f214c33" candidate_uuid: "f2c83d207994493b93f92d1e3e420271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e90f4f716443919ee7b3d1db761808" is_pre_election: true
I20260925 21:56:05.384213 4492 raft_consensus.cc:2454] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2c83d207994493b93f92d1e3e420271 in term 0.
I20260925 21:56:05.384363 4146 leader_election.cc:302] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e90f4f716443919ee7b3d1db761808, f2c83d207994493b93f92d1e3e420271; no voters:
I20260925 21:56:05.384450 4571 raft_consensus.cc:2781] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:05.384478 4571 raft_consensus.cc:482] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:05.384495 4571 raft_consensus.cc:3037] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:05.385279 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c6e24abd6643e0a69c6ace3f214c33" candidate_uuid: "f2c83d207994493b93f92d1e3e420271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" is_pre_election: true
I20260925 21:56:05.385378 4352 raft_consensus.cc:2454] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2c83d207994493b93f92d1e3e420271 in term 0.
I20260925 21:56:05.385416 4571 raft_consensus.cc:504] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.385530 4571 leader_election.cc:288] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [CANDIDATE]: Term 1 election: Requested vote from peers a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765), 75e90f4f716443919ee7b3d1db761808 (127.2.70.195:46663)
I20260925 21:56:05.385629 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c6e24abd6643e0a69c6ace3f214c33" candidate_uuid: "f2c83d207994493b93f92d1e3e420271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e"
I20260925 21:56:05.385699 4352 raft_consensus.cc:3037] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:05.385758 4492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c6e24abd6643e0a69c6ace3f214c33" candidate_uuid: "f2c83d207994493b93f92d1e3e420271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e90f4f716443919ee7b3d1db761808"
I20260925 21:56:05.385818 4492 raft_consensus.cc:3037] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:05.386456 4492 raft_consensus.cc:2454] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2c83d207994493b93f92d1e3e420271 in term 1.
I20260925 21:56:05.386456 4352 raft_consensus.cc:2454] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2c83d207994493b93f92d1e3e420271 in term 1.
I20260925 21:56:05.386705 4143 leader_election.cc:302] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a845e9efd2cc4525ab38d8f81dd0ef1e, f2c83d207994493b93f92d1e3e420271; no voters:
I20260925 21:56:05.383934 4603 leader_election.cc:288] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765), 75e90f4f716443919ee7b3d1db761808 (127.2.70.195:46663)
I20260925 21:56:05.386879 4603 raft_consensus.cc:2781] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:05.386925 4603 raft_consensus.cc:686] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 1 LEADER]: Becoming Leader. State: Replica: f2c83d207994493b93f92d1e3e420271, State: Running, Role: LEADER
I20260925 21:56:05.386965 4603 consensus_queue.cc:237] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:05.387475 4053 catalog_manager.cc:5495] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2c83d207994493b93f92d1e3e420271 (127.2.70.193). New cstate: current_term: 1 leader_uuid: "f2c83d207994493b93f92d1e3e420271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } health_report { overall_health: UNKNOWN } } }
I20260925 21:56:05.396662 4492 raft_consensus.cc:1261] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Refusing update from remote peer f2c83d207994493b93f92d1e3e420271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:05.396703 4352 raft_consensus.cc:1261] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Refusing update from remote peer f2c83d207994493b93f92d1e3e420271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:05.396881 4603 consensus_queue.cc:1035] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:05.397027 4603 consensus_queue.cc:1035] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:05.398728 4589 mvcc.cc:205] Tried to move back new op lower bound from 7333369304663162880 to 7333369304625360896. Current Snapshot: MvccSnapshot[applied={T|T < 7333369304663162880}]
W20260925 21:56:05.402151 4590 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 21:56:06.921700 4053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:06.921805 4053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:06.922744 4053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337: Sending DeleteTablet for 3 replicas of tablet 53c6e24abd6643e0a69c6ace3f214c33
I20260925 21:56:06.923034 4332 tablet_service.cc:1514] Processing DeleteTablet for tablet 53c6e24abd6643e0a69c6ace3f214c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:06 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:47434
I20260925 21:56:06.923084 4192 tablet_service.cc:1514] Processing DeleteTablet for tablet 53c6e24abd6643e0a69c6ace3f214c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:06 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:41582
I20260925 21:56:06.923287 4639 tablet_replica.cc:330] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271: stopping tablet replica
I20260925 21:56:06.923267 4472 tablet_service.cc:1514] Processing DeleteTablet for tablet 53c6e24abd6643e0a69c6ace3f214c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:06 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:35124
I20260925 21:56:06.923347 4639 raft_consensus.cc:2229] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:06.923412 4639 raft_consensus.cc:2258] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:06.923604 4639 ts_tablet_manager.cc:1905] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:06.925189 4639 ts_tablet_manager.cc:1918] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 21:56:06.925243 4639 log.cc:1196] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/wal/wals/53c6e24abd6643e0a69c6ace3f214c33
I20260925 21:56:06.925438 4639 ts_tablet_manager.cc:1939] T 53c6e24abd6643e0a69c6ace3f214c33 P f2c83d207994493b93f92d1e3e420271: Deleting consensus metadata
I20260925 21:56:06.925808 4041 catalog_manager.cc:4841] TS f2c83d207994493b93f92d1e3e420271 (127.2.70.193:34879): tablet 53c6e24abd6643e0a69c6ace3f214c33 (table test-table [id=98d6ae3d8a2749c8afc00b787b6ba386]) successfully deleted
I20260925 21:56:06.926065 4638 tablet_replica.cc:330] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e: stopping tablet replica
I20260925 21:56:06.926136 4638 raft_consensus.cc:2229] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:06.926183 4638 raft_consensus.cc:2258] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:06.926285 4640 tablet_replica.cc:330] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808: stopping tablet replica
I20260925 21:56:06.926347 4640 raft_consensus.cc:2229] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:06.926427 4640 raft_consensus.cc:2258] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:06.926432 4638 ts_tablet_manager.cc:1905] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:06.926658 4640 ts_tablet_manager.cc:1905] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:06.927929 4638 ts_tablet_manager.cc:1918] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 21:56:06.927977 4638 log.cc:1196] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/53c6e24abd6643e0a69c6ace3f214c33
I20260925 21:56:06.928158 4638 ts_tablet_manager.cc:1939] T 53c6e24abd6643e0a69c6ace3f214c33 P a845e9efd2cc4525ab38d8f81dd0ef1e: Deleting consensus metadata
I20260925 21:56:06.928484 4039 catalog_manager.cc:4841] TS a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765): tablet 53c6e24abd6643e0a69c6ace3f214c33 (table test-table [id=98d6ae3d8a2749c8afc00b787b6ba386]) successfully deleted
I20260925 21:56:06.928602 4640 ts_tablet_manager.cc:1918] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 21:56:06.928665 4640 log.cc:1196] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/53c6e24abd6643e0a69c6ace3f214c33
I20260925 21:56:06.928844 4640 ts_tablet_manager.cc:1939] T 53c6e24abd6643e0a69c6ace3f214c33 P 75e90f4f716443919ee7b3d1db761808: Deleting consensus metadata
I20260925 21:56:06.929219 4042 catalog_manager.cc:4841] TS 75e90f4f716443919ee7b3d1db761808 (127.2.70.195:46663): tablet 53c6e24abd6643e0a69c6ace3f214c33 (table test-table [id=98d6ae3d8a2749c8afc00b787b6ba386]) successfully deleted
Sep 25 21:56:06 dist-test-slave-2g0f krb5kdc[4010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373366, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 21:56:06 dist-test-slave-2g0f krb5kdc[4010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373366, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:06.946765 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:06.939536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.2.70.254:36933)
0925 21:56:06.939656 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:06.939658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:06.939694 (+ 36us) server_negotiation.cc:407] Connection header received
0925 21:56:06.939726 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:06.939726 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:06.939737 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:06.939799 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:06.940367 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:06.940903 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:06.941618 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:06.941846 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:06.941896 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:06.944130 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:06.944132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:06.944132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:06.944139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:06.945525 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:06.945938 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:06.945939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:06.945940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:06.946012 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:06.946202 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:06.946203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:06.946203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:06.946317 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:06.946393 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:06.946615 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 21:56:06.946693 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260925 21:56:06.948074 4053 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:06.948225 4053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:06.949465 4192 tablet_service.cc:1467] Processing CreateTablet for tablet bc4c1af9f3d545b8ae08a9b515f02b53 (DEFAULT_TABLE table=test-table [id=2f40b5d1b29d425c90fc11d2ec74ae67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:06.949473 4332 tablet_service.cc:1467] Processing CreateTablet for tablet bc4c1af9f3d545b8ae08a9b515f02b53 (DEFAULT_TABLE table=test-table [id=2f40b5d1b29d425c90fc11d2ec74ae67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:06.949491 4472 tablet_service.cc:1467] Processing CreateTablet for tablet bc4c1af9f3d545b8ae08a9b515f02b53 (DEFAULT_TABLE table=test-table [id=2f40b5d1b29d425c90fc11d2ec74ae67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:06.949599 4332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4c1af9f3d545b8ae08a9b515f02b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:06.949599 4472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4c1af9f3d545b8ae08a9b515f02b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:06.949600 4192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4c1af9f3d545b8ae08a9b515f02b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:06.950557 4649 tablet_bootstrap.cc:492] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271: Bootstrap starting.
I20260925 21:56:06.950879 4651 tablet_bootstrap.cc:492] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808: Bootstrap starting.
I20260925 21:56:06.950955 4649 tablet_bootstrap.cc:654] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:06.951020 4650 tablet_bootstrap.cc:492] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e: Bootstrap starting.
I20260925 21:56:06.951489 4651 tablet_bootstrap.cc:654] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:06.951535 4649 tablet_bootstrap.cc:492] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271: No bootstrap required, opened a new log
I20260925 21:56:06.951577 4649 ts_tablet_manager.cc:1397] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:06.951704 4649 raft_consensus.cc:348] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:06.951786 4649 raft_consensus.cc:374] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:06.951809 4649 raft_consensus.cc:729] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c83d207994493b93f92d1e3e420271, State: Initialized, Role: FOLLOWER
I20260925 21:56:06.951897 4649 consensus_queue.cc:260] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:06.952031 4650 tablet_bootstrap.cc:654] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:06.952127 4649 ts_tablet_manager.cc:1428] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:06.952201 4651 tablet_bootstrap.cc:492] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808: No bootstrap required, opened a new log
I20260925 21:56:06.952260 4651 ts_tablet_manager.cc:1397] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:06.952371 4651 raft_consensus.cc:348] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:06.952445 4651 raft_consensus.cc:374] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:06.952463 4651 raft_consensus.cc:729] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e90f4f716443919ee7b3d1db761808, State: Initialized, Role: FOLLOWER
I20260925 21:56:06.952512 4651 consensus_queue.cc:260] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:06.952602 4651 ts_tablet_manager.cc:1428] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:06.952696 4650 tablet_bootstrap.cc:492] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e: No bootstrap required, opened a new log
I20260925 21:56:06.952777 4650 ts_tablet_manager.cc:1397] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:06.953104 4650 raft_consensus.cc:348] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:06.953187 4650 raft_consensus.cc:374] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:06.953208 4650 raft_consensus.cc:729] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a845e9efd2cc4525ab38d8f81dd0ef1e, State: Initialized, Role: FOLLOWER
I20260925 21:56:06.953246 4650 consensus_queue.cc:260] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:06.953480 4650 ts_tablet_manager.cc:1428] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 21:56:07.003599 4655 raft_consensus.cc:482] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:07.003715 4655 raft_consensus.cc:504] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:07.004076 4655 leader_election.cc:288] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2c83d207994493b93f92d1e3e420271 (127.2.70.193:34879), 75e90f4f716443919ee7b3d1db761808 (127.2.70.195:46663)
I20260925 21:56:07.006929 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:07.004386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:51473 (local address 127.2.70.195:46663)
0925 21:56:07.004515 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:07.004517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:07.004527 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:07.004600 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:07.004600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:07.004611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:07.004671 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:07.004826 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.005304 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.006248 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.006607 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.006668 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:07.006679 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:07.006817 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 21:56:07.006857 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260925 21:56:07.006934 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:07.004285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:46663 (local address 127.2.70.194:51473)
0925 21:56:07.004470 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:07.004483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:07.004543 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:07.004681 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:07.004682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:07.004684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:07.004759 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:07.004761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.005374 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.005375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:07.006144 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:07.006147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.006680 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.006681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:07.006753 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:07.006757 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:07.006801 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 21:56:07.006846 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":112,"threads_started":1}
I20260925 21:56:07.007145 4492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4c1af9f3d545b8ae08a9b515f02b53" candidate_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e90f4f716443919ee7b3d1db761808" is_pre_election: true
I20260925 21:56:07.007192 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:07.004310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:34879 (local address 127.2.70.194:41423)
0925 21:56:07.004479 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:07.004489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:07.004547 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:07.004747 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:07.004748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:07.004750 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:07.004807 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:07.004809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.005415 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.005416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:07.006406 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:07.006408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.006940 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.006940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:07.007011 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:07.007015 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:07.007064 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 21:56:07.007107 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":105,"threads_started":1}
I20260925 21:56:07.007230 4492 raft_consensus.cc:2454] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a845e9efd2cc4525ab38d8f81dd0ef1e in term 0.
I20260925 21:56:07.007262 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:07.004380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:41423 (local address 127.2.70.193:34879)
0925 21:56:07.004532 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:07.004534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:07.004543 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:07.004600 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:07.004600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:07.004611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:07.004671 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:07.004878 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.005347 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.006500 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.006872 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.006936 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:07.006945 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:07.007134 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 21:56:07.007196 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260925 21:56:07.007416 4286 leader_election.cc:302] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e90f4f716443919ee7b3d1db761808, a845e9efd2cc4525ab38d8f81dd0ef1e; no voters:
I20260925 21:56:07.007426 4212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4c1af9f3d545b8ae08a9b515f02b53" candidate_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c83d207994493b93f92d1e3e420271" is_pre_election: true
I20260925 21:56:07.007508 4212 raft_consensus.cc:2454] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a845e9efd2cc4525ab38d8f81dd0ef1e in term 0.
I20260925 21:56:07.007550 4655 raft_consensus.cc:2781] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:07.007588 4655 raft_consensus.cc:482] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:07.007606 4655 raft_consensus.cc:3037] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:07.008335 4655 raft_consensus.cc:504] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:07.008455 4655 leader_election.cc:288] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [CANDIDATE]: Term 1 election: Requested vote from peers f2c83d207994493b93f92d1e3e420271 (127.2.70.193:34879), 75e90f4f716443919ee7b3d1db761808 (127.2.70.195:46663)
I20260925 21:56:07.008621 4492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4c1af9f3d545b8ae08a9b515f02b53" candidate_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e90f4f716443919ee7b3d1db761808"
I20260925 21:56:07.008621 4212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4c1af9f3d545b8ae08a9b515f02b53" candidate_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c83d207994493b93f92d1e3e420271"
I20260925 21:56:07.008697 4212 raft_consensus.cc:3037] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:07.008697 4492 raft_consensus.cc:3037] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:07.009402 4492 raft_consensus.cc:2454] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a845e9efd2cc4525ab38d8f81dd0ef1e in term 1.
I20260925 21:56:07.009415 4212 raft_consensus.cc:2454] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a845e9efd2cc4525ab38d8f81dd0ef1e in term 1.
I20260925 21:56:07.009557 4285 leader_election.cc:302] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a845e9efd2cc4525ab38d8f81dd0ef1e, f2c83d207994493b93f92d1e3e420271; no voters:
I20260925 21:56:07.009660 4655 raft_consensus.cc:2781] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:07.009835 4655 raft_consensus.cc:686] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 LEADER]: Becoming Leader. State: Replica: a845e9efd2cc4525ab38d8f81dd0ef1e, State: Running, Role: LEADER
I20260925 21:56:07.009902 4655 consensus_queue.cc:237] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:07.010624 4053 catalog_manager.cc:5495] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e reported cstate change: term changed from 0 to 1, leader changed from <none> to a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194). New cstate: current_term: 1 leader_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } health_report { overall_health: UNKNOWN } } }
I20260925 21:56:07.030073 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:07.027615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.2.70.194:40765)
0925 21:56:07.027777 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:07.027779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:07.027785 (+ 6us) server_negotiation.cc:407] Connection header received
0925 21:56:07.027819 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:07.027819 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:07.027859 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:07.027935 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:07.028040 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.028580 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.029363 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.029632 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.029686 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:07.029803 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:07.029907 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:07.029956 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:07.029971 (+ 15us) server_negotiation.cc:299] Negotiation successful
0925 21:56:07.029986 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260925 21:56:07.034818 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:07.032539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:49181 (local address 127.2.70.195:46663)
0925 21:56:07.032603 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:07.032605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:07.032661 (+ 56us) server_negotiation.cc:407] Connection header received
0925 21:56:07.032769 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:07.032770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:07.032781 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:07.032822 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:07.032947 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.033399 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.034181 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.034371 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.034423 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:07.034596 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:07.034623 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:07.034662 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:07.034720 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 21:56:07.034739 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 21:56:07.034832 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:07.032456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:46663 (local address 127.2.70.194:49181)
0925 21:56:07.032593 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:07.032604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:07.032679 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:07.032830 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:07.032831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:07.032833 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 21:56:07.032892 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:07.032894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.033465 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.033466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:07.034093 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:07.034095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.034433 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.034434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:07.034506 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:07.034519 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 21:56:07.034669 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 21:56:07.034671 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:07.034706 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 21:56:07.034747 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
I20260925 21:56:07.036870 4492 raft_consensus.cc:1261] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Refusing update from remote peer a845e9efd2cc4525ab38d8f81dd0ef1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:07.036872 4212 raft_consensus.cc:1261] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Refusing update from remote peer a845e9efd2cc4525ab38d8f81dd0ef1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:07.037098 4660 consensus_queue.cc:1035] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:07.037220 4655 consensus_queue.cc:1035] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:07.043757 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:07.041595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:58257 (local address 127.2.70.194:40765)
0925 21:56:07.041628 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:07.041630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:07.041731 (+ 101us) server_negotiation.cc:407] Connection header received
0925 21:56:07.041800 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:07.041801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:07.041813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:07.041857 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:07.041945 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.042391 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.043126 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.043334 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.043391 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:07.043524 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:07.043556 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:07.043613 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:07.043664 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 21:56:07.043683 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 21:56:07.043780 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:07.041556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:40765 (local address 127.2.70.195:58257)
0925 21:56:07.041671 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:07.041680 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:07.041731 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:07.041862 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:07.041863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:07.041864 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 21:56:07.041908 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:07.041909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.042446 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.042447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:07.043064 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:07.043066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.043398 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.043399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:07.043452 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:07.043457 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 21:56:07.043619 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 21:56:07.043620 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 21:56:07.043660 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 21:56:07.043700 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":74,"threads_started":1}
I20260925 21:56:07.050004 4052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:07.050102 4052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:07.050765 4052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337: Sending DeleteTablet for 3 replicas of tablet bc4c1af9f3d545b8ae08a9b515f02b53
I20260925 21:56:07.050998 4472 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4c1af9f3d545b8ae08a9b515f02b53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:07 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:35124
I20260925 21:56:07.050999 4192 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4c1af9f3d545b8ae08a9b515f02b53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:07 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:41582
I20260925 21:56:07.051095 4639 tablet_replica.cc:330] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271: stopping tablet replica
I20260925 21:56:07.051095 4640 tablet_replica.cc:330] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808: stopping tablet replica
I20260925 21:56:07.051143 4639 raft_consensus.cc:2229] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:07.051146 4640 raft_consensus.cc:2229] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:07.051177 4639 raft_consensus.cc:2258] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:07.051179 4640 raft_consensus.cc:2258] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:07.051496 4640 ts_tablet_manager.cc:1905] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:07.051544 4639 ts_tablet_manager.cc:1905] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:07.053193 4640 ts_tablet_manager.cc:1918] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:07.053259 4640 log.cc:1196] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/bc4c1af9f3d545b8ae08a9b515f02b53
I20260925 21:56:07.053288 4332 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4c1af9f3d545b8ae08a9b515f02b53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:07 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:47434
I20260925 21:56:07.053378 4638 tablet_replica.cc:330] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e: stopping tablet replica
I20260925 21:56:07.053427 4638 raft_consensus.cc:2229] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:07.053463 4640 ts_tablet_manager.cc:1939] T bc4c1af9f3d545b8ae08a9b515f02b53 P 75e90f4f716443919ee7b3d1db761808: Deleting consensus metadata
I20260925 21:56:07.053520 4638 raft_consensus.cc:2258] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:07.053511 4052 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 21:56:07.053642 4639 ts_tablet_manager.cc:1918] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:07.053685 4639 log.cc:1196] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-0/wal/wals/bc4c1af9f3d545b8ae08a9b515f02b53
W20260925 21:56:07.053673 4052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:07.053875 4639 ts_tablet_manager.cc:1939] T bc4c1af9f3d545b8ae08a9b515f02b53 P f2c83d207994493b93f92d1e3e420271: Deleting consensus metadata
I20260925 21:56:07.054068 4042 catalog_manager.cc:4841] TS 75e90f4f716443919ee7b3d1db761808 (127.2.70.195:46663): tablet bc4c1af9f3d545b8ae08a9b515f02b53 (table test-table [id=2f40b5d1b29d425c90fc11d2ec74ae67]) successfully deleted
I20260925 21:56:07.054157 4041 catalog_manager.cc:4841] TS f2c83d207994493b93f92d1e3e420271 (127.2.70.193:34879): tablet bc4c1af9f3d545b8ae08a9b515f02b53 (table test-table [id=2f40b5d1b29d425c90fc11d2ec74ae67]) successfully deleted
I20260925 21:56:07.054418 4638 ts_tablet_manager.cc:1905] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:07.055117 4472 tablet_service.cc:1467] Processing CreateTablet for tablet 7180eac3651341fdbb6ee7712b69813e (DEFAULT_TABLE table=test-table [id=09d08479cdbf4189bdc7c399e4678a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:07.055119 4192 tablet_service.cc:1467] Processing CreateTablet for tablet 7180eac3651341fdbb6ee7712b69813e (DEFAULT_TABLE table=test-table [id=09d08479cdbf4189bdc7c399e4678a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:07.055223 4472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7180eac3651341fdbb6ee7712b69813e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:07.055229 4192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7180eac3651341fdbb6ee7712b69813e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:07.056016 4332 tablet_service.cc:1467] Processing CreateTablet for tablet 7180eac3651341fdbb6ee7712b69813e (DEFAULT_TABLE table=test-table [id=09d08479cdbf4189bdc7c399e4678a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:07.056270 4638 ts_tablet_manager.cc:1918] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:07.056316 4638 log.cc:1196] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/bc4c1af9f3d545b8ae08a9b515f02b53
I20260925 21:56:07.056512 4638 ts_tablet_manager.cc:1939] T bc4c1af9f3d545b8ae08a9b515f02b53 P a845e9efd2cc4525ab38d8f81dd0ef1e: Deleting consensus metadata
I20260925 21:56:07.056618 4651 tablet_bootstrap.cc:492] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808: Bootstrap starting.
I20260925 21:56:07.056631 4332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7180eac3651341fdbb6ee7712b69813e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:07.056942 4651 tablet_bootstrap.cc:654] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:07.057507 4651 tablet_bootstrap.cc:492] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808: No bootstrap required, opened a new log
I20260925 21:56:07.057550 4651 ts_tablet_manager.cc:1397] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:07.057672 4650 tablet_bootstrap.cc:492] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e: Bootstrap starting.
I20260925 21:56:07.057686 4649 tablet_bootstrap.cc:492] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271: Bootstrap starting.
I20260925 21:56:07.057709 4651 raft_consensus.cc:348] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:07.057780 4651 raft_consensus.cc:374] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:07.057799 4651 raft_consensus.cc:729] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e90f4f716443919ee7b3d1db761808, State: Initialized, Role: FOLLOWER
I20260925 21:56:07.057847 4651 consensus_queue.cc:260] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:07.057936 4651 ts_tablet_manager.cc:1428] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:07.057988 4649 tablet_bootstrap.cc:654] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:07.058051 4650 tablet_bootstrap.cc:654] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:07.058701 4039 catalog_manager.cc:4841] TS a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765): tablet bc4c1af9f3d545b8ae08a9b515f02b53 (table test-table [id=2f40b5d1b29d425c90fc11d2ec74ae67]) successfully deleted
I20260925 21:56:07.058746 4650 tablet_bootstrap.cc:492] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e: No bootstrap required, opened a new log
I20260925 21:56:07.058785 4650 ts_tablet_manager.cc:1397] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:07.058940 4650 raft_consensus.cc:348] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:07.059020 4650 raft_consensus.cc:374] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:07.059041 4650 raft_consensus.cc:729] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a845e9efd2cc4525ab38d8f81dd0ef1e, State: Initialized, Role: FOLLOWER
I20260925 21:56:07.059079 4650 consensus_queue.cc:260] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:07.059201 4650 ts_tablet_manager.cc:1428] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:07.059636 4649 tablet_bootstrap.cc:492] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271: No bootstrap required, opened a new log
I20260925 21:56:07.059689 4649 ts_tablet_manager.cc:1397] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:07.059799 4649 raft_consensus.cc:348] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:07.059899 4649 raft_consensus.cc:374] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:07.059921 4649 raft_consensus.cc:729] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c83d207994493b93f92d1e3e420271, State: Initialized, Role: FOLLOWER
I20260925 21:56:07.059966 4649 consensus_queue.cc:260] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:07.060055 4649 ts_tablet_manager.cc:1428] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:07.169516 4624 raft_consensus.cc:482] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:07.169682 4624 raft_consensus.cc:504] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:07.169870 4624 leader_election.cc:288] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2c83d207994493b93f92d1e3e420271 (127.2.70.193:34879), a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765)
I20260925 21:56:07.170109 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7180eac3651341fdbb6ee7712b69813e" candidate_uuid: "75e90f4f716443919ee7b3d1db761808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" is_pre_election: true
I20260925 21:56:07.170126 4212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7180eac3651341fdbb6ee7712b69813e" candidate_uuid: "75e90f4f716443919ee7b3d1db761808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c83d207994493b93f92d1e3e420271" is_pre_election: true
I20260925 21:56:07.170198 4352 raft_consensus.cc:2454] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e90f4f716443919ee7b3d1db761808 in term 0.
I20260925 21:56:07.170235 4212 raft_consensus.cc:2454] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e90f4f716443919ee7b3d1db761808 in term 0.
I20260925 21:56:07.170361 4423 leader_election.cc:302] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e90f4f716443919ee7b3d1db761808, a845e9efd2cc4525ab38d8f81dd0ef1e; no voters:
I20260925 21:56:07.170468 4624 raft_consensus.cc:2781] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:07.170508 4624 raft_consensus.cc:482] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:07.170526 4624 raft_consensus.cc:3037] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:07.171281 4624 raft_consensus.cc:504] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:07.171422 4624 leader_election.cc:288] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [CANDIDATE]: Term 1 election: Requested vote from peers f2c83d207994493b93f92d1e3e420271 (127.2.70.193:34879), a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765)
I20260925 21:56:07.171633 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7180eac3651341fdbb6ee7712b69813e" candidate_uuid: "75e90f4f716443919ee7b3d1db761808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e"
I20260925 21:56:07.171634 4212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7180eac3651341fdbb6ee7712b69813e" candidate_uuid: "75e90f4f716443919ee7b3d1db761808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c83d207994493b93f92d1e3e420271"
I20260925 21:56:07.171715 4352 raft_consensus.cc:3037] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:07.171716 4212 raft_consensus.cc:3037] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:07.172490 4352 raft_consensus.cc:2454] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e90f4f716443919ee7b3d1db761808 in term 1.
I20260925 21:56:07.172490 4212 raft_consensus.cc:2454] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e90f4f716443919ee7b3d1db761808 in term 1.
I20260925 21:56:07.172670 4425 leader_election.cc:302] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e90f4f716443919ee7b3d1db761808, f2c83d207994493b93f92d1e3e420271; no voters:
I20260925 21:56:07.172780 4624 raft_consensus.cc:2781] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:07.172828 4624 raft_consensus.cc:686] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 1 LEADER]: Becoming Leader. State: Replica: 75e90f4f716443919ee7b3d1db761808, State: Running, Role: LEADER
I20260925 21:56:07.172876 4624 consensus_queue.cc:237] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } }
I20260925 21:56:07.173483 4052 catalog_manager.cc:5495] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75e90f4f716443919ee7b3d1db761808 (127.2.70.195). New cstate: current_term: 1 leader_uuid: "75e90f4f716443919ee7b3d1db761808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75e90f4f716443919ee7b3d1db761808" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46663 } health_report { overall_health: HEALTHY } } }
I20260925 21:56:07.180817 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:07.178439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.2.70.195:46663)
0925 21:56:07.178486 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:07.178488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:07.178496 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:07.178650 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:07.178651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:07.178661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:07.178705 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:07.178805 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.179286 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.180203 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.180404 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.180461 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:07.180584 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:07.180618 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:07.180657 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:07.180715 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 21:56:07.180736 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 21:56:07.185352 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:07.183272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:52917 (local address 127.2.70.195:46663)
0925 21:56:07.183319 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:07.183320 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:07.183326 (+ 6us) server_negotiation.cc:407] Connection header received
0925 21:56:07.183340 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:07.183341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:07.183347 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:07.183388 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:07.183488 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.184029 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.184740 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.184939 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.184992 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:07.185111 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:07.185145 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:07.185185 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:07.185229 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 21:56:07.185246 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 21:56:07.185417 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:07.183145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:46663 (local address 127.2.70.195:52917)
0925 21:56:07.183208 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:07.183220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:07.183257 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:07.183397 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:07.183398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:07.183399 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 21:56:07.183435 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:07.183437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.184118 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.184119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:07.184667 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:07.184669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:07.184995 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:07.184996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:07.185046 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:07.185052 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 21:56:07.185185 (+ 133us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 21:56:07.185187 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:07.185243 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 21:56:07.185285 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 21:56:07.187209 4212 raft_consensus.cc:1261] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Refusing update from remote peer 75e90f4f716443919ee7b3d1db761808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:07.187343 4352 raft_consensus.cc:1261] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Refusing update from remote peer 75e90f4f716443919ee7b3d1db761808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:07.187392 4624 consensus_queue.cc:1035] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2c83d207994493b93f92d1e3e420271" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:07.187561 4624 consensus_queue.cc:1035] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a845e9efd2cc4525ab38d8f81dd0ef1e" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 21:56:07.191915 4617 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 21:56:08.717777 4054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:08.717885 4054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:08.718685 4054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33bf6a147986440d8890fa202d261337: Sending DeleteTablet for 3 replicas of tablet 7180eac3651341fdbb6ee7712b69813e
I20260925 21:56:08.718931 4471 tablet_service.cc:1514] Processing DeleteTablet for tablet 7180eac3651341fdbb6ee7712b69813e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:08 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:35124
I20260925 21:56:08.718984 4332 tablet_service.cc:1514] Processing DeleteTablet for tablet 7180eac3651341fdbb6ee7712b69813e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:08 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:47434
I20260925 21:56:08.718993 4192 tablet_service.cc:1514] Processing DeleteTablet for tablet 7180eac3651341fdbb6ee7712b69813e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:08 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:41582
I20260925 21:56:08.719169 4702 tablet_replica.cc:330] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271: stopping tablet replica
I20260925 21:56:08.719177 4703 tablet_replica.cc:330] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e: stopping tablet replica
I20260925 21:56:08.719229 4703 raft_consensus.cc:2229] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:08.719229 4702 raft_consensus.cc:2229] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:08.719272 4702 raft_consensus.cc:2258] T 7180eac3651341fdbb6ee7712b69813e P f2c83d207994493b93f92d1e3e420271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:08.719277 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 4127
I20260925 21:56:08.719293 4703 raft_consensus.cc:2258] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:08.720252 4703 ts_tablet_manager.cc:1905] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:08.721127 4701 tablet_replica.cc:330] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808: stopping tablet replica
I20260925 21:56:08.721199 4701 raft_consensus.cc:2229] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:08.721319 4701 raft_consensus.cc:2258] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:08.721616 4701 ts_tablet_manager.cc:1905] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:08.721987 4703 ts_tablet_manager.cc:1918] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 21:56:08.722043 4703 log.cc:1196] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/7180eac3651341fdbb6ee7712b69813e
I20260925 21:56:08.722235 4703 ts_tablet_manager.cc:1939] T 7180eac3651341fdbb6ee7712b69813e P a845e9efd2cc4525ab38d8f81dd0ef1e: Deleting consensus metadata
I20260925 21:56:08.722684 4039 catalog_manager.cc:4841] TS a845e9efd2cc4525ab38d8f81dd0ef1e (127.2.70.194:40765): tablet 7180eac3651341fdbb6ee7712b69813e (table test-table [id=09d08479cdbf4189bdc7c399e4678a0e]) successfully deleted
I20260925 21:56:08.723162 4701 ts_tablet_manager.cc:1918] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 21:56:08.723223 4701 log.cc:1196] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/7180eac3651341fdbb6ee7712b69813e
I20260925 21:56:08.723434 4701 ts_tablet_manager.cc:1939] T 7180eac3651341fdbb6ee7712b69813e P 75e90f4f716443919ee7b3d1db761808: Deleting consensus metadata
I20260925 21:56:08.723908 4042 catalog_manager.cc:4841] TS 75e90f4f716443919ee7b3d1db761808 (127.2.70.195:46663): tablet 7180eac3651341fdbb6ee7712b69813e (table test-table [id=09d08479cdbf4189bdc7c399e4678a0e]) successfully deleted
W20260925 21:56:08.725804 4041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:34879 (error 108)
I20260925 21:56:08.725967 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 4267
I20260925 21:56:08.726855 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:08.726646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:34879 (local address 127.0.0.1:41594)
0925 21:56:08.726770 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:08.726814 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:34879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
W20260925 21:56:08.727007 4041 catalog_manager.cc:4584] TS f2c83d207994493b93f92d1e3e420271 (127.2.70.193:34879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7180eac3651341fdbb6ee7712b69813e: Network error: Client connection negotiation failed: client connection to 127.2.70.193:34879: connect: Connection refused (error 111)
I20260925 21:56:08.731310 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 4407
I20260925 21:56:08.736091 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 4023
2026-09-25T21:56:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5437 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:08 dist-test-slave-2g0f krb5kdc[4708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:08 dist-test-slave-2g0f krb5kdc[4708](info): set up 2 sockets
Sep 25 21:56:08 dist-test-slave-2g0f krb5kdc[4708](info): commencing operation
krb5kdc: starting...
W20260925 21:56:09.703637 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:09 dist-test-slave-2g0f krb5kdc[4708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:56:09.825032 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:39897
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:39897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:09.906311 4721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:09.906558 4721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:09.906590 4721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:09.907797 4721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:09.907868 4721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:09.907883 4721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:09.907895 4721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:09.907914 4721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:09.909464 4721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:39897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:09.909703 4721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:09.909895 4721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:09.914954 4729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:09.915166 4721 server_base.cc:1034] running on GCE node
W20260925 21:56:09.914954 4727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:09.915027 4726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:09.915593 4721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:09.915937 4721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:09.917107 4721 hybrid_clock.cc:648] HybridClock initialized: now 1790373369917089 us; error 29 us; skew 500 ppm
Sep 25 21:56:09 dist-test-slave-2g0f krb5kdc[4708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:09.938553 4721 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:56:09.939106 4721 webserver.cc:460] Webserver started at http://127.2.70.254:41299/ using document root <none> and password file <none>
I20260925 21:56:09.939334 4721 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:09.939379 4721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:09.939487 4721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:09.940661 4721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "46327fd54df44537910ec1d549026591"
format_stamp: "Formatted at 2026-09-25 21:56:09 on dist-test-slave-2g0f"
server_key: "42706da9bfa7634acd0f36548bd5d491"
server_key_iv: "f5981c033bb26fbb8b3a7fe44ee6426b"
server_key_version: "encryptionkey@0"
I20260925 21:56:09.941010 4721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "46327fd54df44537910ec1d549026591"
format_stamp: "Formatted at 2026-09-25 21:56:09 on dist-test-slave-2g0f"
server_key: "42706da9bfa7634acd0f36548bd5d491"
server_key_iv: "f5981c033bb26fbb8b3a7fe44ee6426b"
server_key_version: "encryptionkey@0"
I20260925 21:56:09.942349 4721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 21:56:09.943130 4736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:09.943336 4721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:56:09.943429 4721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "46327fd54df44537910ec1d549026591"
format_stamp: "Formatted at 2026-09-25 21:56:09 on dist-test-slave-2g0f"
server_key: "42706da9bfa7634acd0f36548bd5d491"
server_key_iv: "f5981c033bb26fbb8b3a7fe44ee6426b"
server_key_version: "encryptionkey@0"
I20260925 21:56:09.943487 4721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:09.959662 4721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:09.962229 4721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:09.962360 4721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:09.966432 4721 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:39897
I20260925 21:56:09.966562 4788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:39897 every 8 connection(s)
I20260925 21:56:09.966873 4721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:09.968209 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:09.970283 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 4721
I20260925 21:56:09.970371 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:09.970522 2331 external_mini_cluster.cc:1428] Setting key 685a4783958d4960e7251c7ea1fffebb
I20260925 21:56:09.971692 4789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591: Bootstrap starting.
I20260925 21:56:09.972349 4789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:09.972656 4789 log.cc:824] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:09.973335 4789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591: No bootstrap required, opened a new log
I20260925 21:56:09.975189 4789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46327fd54df44537910ec1d549026591" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39897 } }
I20260925 21:56:09.975340 4789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:09.975373 4789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46327fd54df44537910ec1d549026591, State: Initialized, Role: FOLLOWER
I20260925 21:56:09.975518 4789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46327fd54df44537910ec1d549026591" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39897 } }
I20260925 21:56:09.975598 4789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:09.975639 4789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:09.975687 4789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 21:56:09 dist-test-slave-2g0f krb5kdc[4708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:09.981966 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:09.971565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.2.70.254:39897)
0925 21:56:09.971801 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:09.971814 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:09.971881 (+ 67us) server_negotiation.cc:407] Connection header received
0925 21:56:09.972374 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:09.972381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:09.972515 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:09.972674 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:09.973412 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:09.974378 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:09.975160 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:09.975683 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:09.975813 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:09.978560 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:09.978565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:09.978565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:09.978579 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:09.980375 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:09.980769 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:09.980772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:09.980772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:09.980866 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:09.981064 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:09.981065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:09.981065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:09.981351 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:09.981480 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:09.981604 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 21:56:09.981766 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":97,"threads_started":1}
I20260925 21:56:09.986508 4789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46327fd54df44537910ec1d549026591" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39897 } }
I20260925 21:56:09.986817 4789 leader_election.cc:302] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46327fd54df44537910ec1d549026591; no voters:
I20260925 21:56:09.987087 4789 leader_election.cc:288] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:09.987254 4794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:09.987358 4789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:09.987409 4794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [term 1 LEADER]: Becoming Leader. State: Replica: 46327fd54df44537910ec1d549026591, State: Running, Role: LEADER
I20260925 21:56:09.987514 4794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46327fd54df44537910ec1d549026591" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39897 } }
I20260925 21:56:09.987959 4796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46327fd54df44537910ec1d549026591" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46327fd54df44537910ec1d549026591" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39897 } } }
I20260925 21:56:09.988047 4796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:09.987947 4794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46327fd54df44537910ec1d549026591. Latest consensus state: current_term: 1 leader_uuid: "46327fd54df44537910ec1d549026591" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46327fd54df44537910ec1d549026591" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39897 } } }
I20260925 21:56:09.988234 4794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:09.988258 4803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:09.988864 4803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:09.992367 4803 catalog_manager.cc:1347] Generated new cluster ID: c42532bdaae948b5b5868ba2e77ce4e1
I20260925 21:56:09.992455 4803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:10.000864 4803 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:10.001467 4803 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:10.005949 4803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46327fd54df44537910ec1d549026591: Generated new TSK 0
I20260925 21:56:10.006181 4803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
W20260925 21:56:11.187991 2331 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.70.193: real 1.175s user 0.001s sys 0.002s
I20260925 21:56:11.188391 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39897
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:11.278362 4815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:11.278610 4815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:11.278638 4815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:11.280198 4815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:11.280270 4815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:11.280303 4815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:11.282073 4815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:11.282346 4815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:11.282588 4815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:11.288183 4823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:11.288300 4821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:11.288285 4820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:11.289077 4815 server_base.cc:1034] running on GCE node
I20260925 21:56:11.289305 4815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:11.289582 4815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:11.290757 4815 hybrid_clock.cc:648] HybridClock initialized: now 1790373371290736 us; error 39 us; skew 500 ppm
Sep 25 21:56:11 dist-test-slave-2g0f krb5kdc[4708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373371, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:11.293596 4815 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:56:11.294101 4815 webserver.cc:460] Webserver started at http://127.2.70.193:44679/ using document root <none> and password file <none>
I20260925 21:56:11.294317 4815 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:11.294361 4815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:11.294502 4815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:11.295593 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "25b99c3c84c444e28bbd19cc60168c35"
format_stamp: "Formatted at 2026-09-25 21:56:11 on dist-test-slave-2g0f"
server_key: "a70cb412c4428b490a38ad81c5309ed1"
server_key_iv: "77e8a2b49c4369028cd72f52ab17a795"
server_key_version: "encryptionkey@0"
I20260925 21:56:11.295964 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "25b99c3c84c444e28bbd19cc60168c35"
format_stamp: "Formatted at 2026-09-25 21:56:11 on dist-test-slave-2g0f"
server_key: "a70cb412c4428b490a38ad81c5309ed1"
server_key_iv: "77e8a2b49c4369028cd72f52ab17a795"
server_key_version: "encryptionkey@0"
I20260925 21:56:11.297308 4815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:11.298249 4830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:11.298499 4815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:11.298596 4815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "25b99c3c84c444e28bbd19cc60168c35"
format_stamp: "Formatted at 2026-09-25 21:56:11 on dist-test-slave-2g0f"
server_key: "a70cb412c4428b490a38ad81c5309ed1"
server_key_iv: "77e8a2b49c4369028cd72f52ab17a795"
server_key_version: "encryptionkey@0"
I20260925 21:56:11.298676 4815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:11.329478 4815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:11.332180 4815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:11.332329 4815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:11.332578 4815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:11.332919 4815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:11.332959 4815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:11.332993 4815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:11.333009 4815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:11.339741 4815 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:40111
I20260925 21:56:11.339929 4943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:40111 every 8 connection(s)
I20260925 21:56:11.340160 4815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:11.344038 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 4815
I20260925 21:56:11.344128 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:11.344298 2331 external_mini_cluster.cc:1428] Setting key 8d269e38ee68a163201287abef1ab4fb
Sep 25 21:56:11 dist-test-slave-2g0f krb5kdc[4708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373371, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:11.351037 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:11.340895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:41433 (local address 127.2.70.254:39897)
0925 21:56:11.341111 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:11.341114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:11.341870 (+ 756us) server_negotiation.cc:407] Connection header received
0925 21:56:11.341905 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:11.341907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:11.341920 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:11.341984 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:11.343215 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:11.344066 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:11.345619 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:11.345930 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:11.345995 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:11.348228 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:11.348229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:11.348230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:11.348237 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:11.349636 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:11.350058 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:11.350059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:11.350059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:11.350107 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:11.350342 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:11.350342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:11.350343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:11.350481 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:11.350530 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:11.350873 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 21:56:11.350959 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":45,"threads_started":1}
I20260925 21:56:11.351246 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:11.340922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:39897 (local address 127.2.70.193:41433)
0925 21:56:11.341066 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:11.341107 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:11.341732 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:11.342095 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:11.342100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:11.342902 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:11.343119 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:11.343124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:11.344216 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:11.344217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:11.345523 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:11.345527 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:11.346003 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:11.346004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:11.346137 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:11.346620 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:11.346630 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:11.348123 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:11.349712 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:11.349718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:11.349721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:11.349993 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:11.350158 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:11.350159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:11.350159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:11.350287 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:11.350539 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:11.350545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:11.350706 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 21:56:11.350825 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 21:56:11.350995 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260925 21:56:11.351725 4944 heartbeater.cc:347] Connected to a master server at 127.2.70.254:39897
I20260925 21:56:11.351825 4944 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:11.352037 4944 heartbeater.cc:511] Master 127.2.70.254:39897 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
I20260925 21:56:11.352555 4753 ts_manager.cc:194] Registered new tserver with Master: 25b99c3c84c444e28bbd19cc60168c35 (127.2.70.193:40111)
I20260925 21:56:11.353155 4753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:41433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
I20260925 21:56:12.354110 4944 heartbeater.cc:503] Master 127.2.70.254:39897 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
W20260925 21:56:12.521963 2331 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.70.194: real 1.177s user 0.000s sys 0.003s
I20260925 21:56:12.522287 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39897
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:12.607300 4950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:12.607515 4950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:12.607551 4950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:12.608829 4950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:12.608882 4950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:12.608920 4950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:12.610459 4950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:12.610710 4950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:12.610935 4950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:12.616230 4955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:12.616230 4958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:12.616374 4950 server_base.cc:1034] running on GCE node
W20260925 21:56:12.616230 4956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:12.616696 4950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:12.617002 4950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:12.618144 4950 hybrid_clock.cc:648] HybridClock initialized: now 1790373372618131 us; error 26 us; skew 500 ppm
Sep 25 21:56:12 dist-test-slave-2g0f krb5kdc[4708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373372, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:12.620601 4950 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:56:12.621078 4950 webserver.cc:460] Webserver started at http://127.2.70.194:46607/ using document root <none> and password file <none>
I20260925 21:56:12.621308 4950 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:12.621356 4950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:12.621482 4950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:12.622514 4950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "61528a4d8ce64d0798dc637d3c5e59e9"
format_stamp: "Formatted at 2026-09-25 21:56:12 on dist-test-slave-2g0f"
server_key: "57241987877c4d7c25d77cf2602a7dc8"
server_key_iv: "9d2ae734b4140bf4c5d3ce31ec14cdc2"
server_key_version: "encryptionkey@0"
I20260925 21:56:12.622840 4950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "61528a4d8ce64d0798dc637d3c5e59e9"
format_stamp: "Formatted at 2026-09-25 21:56:12 on dist-test-slave-2g0f"
server_key: "57241987877c4d7c25d77cf2602a7dc8"
server_key_iv: "9d2ae734b4140bf4c5d3ce31ec14cdc2"
server_key_version: "encryptionkey@0"
I20260925 21:56:12.624270 4950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 21:56:12.624961 4965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:12.625113 4950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:12.625192 4950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "61528a4d8ce64d0798dc637d3c5e59e9"
format_stamp: "Formatted at 2026-09-25 21:56:12 on dist-test-slave-2g0f"
server_key: "57241987877c4d7c25d77cf2602a7dc8"
server_key_iv: "9d2ae734b4140bf4c5d3ce31ec14cdc2"
server_key_version: "encryptionkey@0"
I20260925 21:56:12.625257 4950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:12.649828 4950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:12.652017 4950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:12.652165 4950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:12.652400 4950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:12.652727 4950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:12.652760 4950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:12.652788 4950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:12.652822 4950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:12.658457 4950 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:46787
I20260925 21:56:12.658497 5078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:46787 every 8 connection(s)
I20260925 21:56:12.658815 4950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
Sep 25 21:56:12 dist-test-slave-2g0f krb5kdc[4708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373372, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:12.667928 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 4950
I20260925 21:56:12.668040 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:12.668242 2331 external_mini_cluster.cc:1428] Setting key 7d0e33adad5667560ffd56d84a0057e2
I20260925 21:56:12.669674 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:12.659596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:48745 (local address 127.2.70.254:39897)
0925 21:56:12.659715 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:12.659717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:12.659863 (+ 146us) server_negotiation.cc:407] Connection header received
0925 21:56:12.660430 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:12.660432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:12.660448 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:12.660534 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:12.661592 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:12.662171 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:12.663308 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:12.663538 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:12.663603 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:12.666004 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:12.666006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:12.666006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:12.666014 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:12.667590 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:12.668138 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:12.668140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:12.668141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:12.668224 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:12.668641 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:12.668644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:12.668645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:12.668918 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:12.668992 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:12.669430 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 21:56:12.669578 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":42,"threads_started":1}
I20260925 21:56:12.669736 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:12.659579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:39897 (local address 127.2.70.194:48745)
0925 21:56:12.659732 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:12.659769 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:12.660318 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:12.660620 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:12.660624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:12.661194 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:12.661483 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:12.661489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:12.662262 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:12.662263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:12.663215 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:12.663218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:12.663631 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:12.663632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:12.663752 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:12.664382 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:12.664391 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:12.665872 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:12.667704 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:12.667715 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:12.667717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:12.668032 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:12.668298 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:12.668299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:12.668300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:12.668446 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:12.669028 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:12.669037 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:12.669232 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 21:56:12.669353 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 21:56:12.669530 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260925 21:56:12.670384 5079 heartbeater.cc:347] Connected to a master server at 127.2.70.254:39897
I20260925 21:56:12.670497 5079 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:12.670738 5079 heartbeater.cc:511] Master 127.2.70.254:39897 requested a full tablet report, sending...
I20260925 21:56:12.671401 4753 ts_manager.cc:194] Registered new tserver with Master: 61528a4d8ce64d0798dc637d3c5e59e9 (127.2.70.194:46787)
I20260925 21:56:12.671859 4753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:48745
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
W20260925 21:56:13.604453 2331 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.70.195: real 0.935s user 0.003s sys 0.000s
I20260925 21:56:13.604804 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39897
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:56:13.672940 5079 heartbeater.cc:503] Master 127.2.70.254:39897 was elected leader, sending a full tablet report...
W20260925 21:56:13.686126 5085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:13.686357 5085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:13.686381 5085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:13.687752 5085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:13.687800 5085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:13.687852 5085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:13.689508 5085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:13.689724 5085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:13.689924 5085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:56:13.695161 5085 server_base.cc:1034] running on GCE node
W20260925 21:56:13.695133 5091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:13.695122 5090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:13.695133 5093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:13.695595 5085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:13.695886 5085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:13.697064 5085 hybrid_clock.cc:648] HybridClock initialized: now 1790373373697049 us; error 31 us; skew 500 ppm
Sep 25 21:56:13 dist-test-slave-2g0f krb5kdc[4708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373373, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:13.699981 5085 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:56:13.700479 5085 webserver.cc:460] Webserver started at http://127.2.70.195:46707/ using document root <none> and password file <none>
I20260925 21:56:13.700721 5085 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:13.700768 5085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:13.700899 5085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:13.702056 5085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "3370421101d54dd5b0110b8e24b2b80b"
format_stamp: "Formatted at 2026-09-25 21:56:13 on dist-test-slave-2g0f"
server_key: "58aaa65ca39bcc8d61b5220541f78d82"
server_key_iv: "d1c3a9226751b55349c7172a89b5c30d"
server_key_version: "encryptionkey@0"
I20260925 21:56:13.702401 5085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "3370421101d54dd5b0110b8e24b2b80b"
format_stamp: "Formatted at 2026-09-25 21:56:13 on dist-test-slave-2g0f"
server_key: "58aaa65ca39bcc8d61b5220541f78d82"
server_key_iv: "d1c3a9226751b55349c7172a89b5c30d"
server_key_version: "encryptionkey@0"
I20260925 21:56:13.703711 5085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 21:56:13.704535 5100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:13.704713 5085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:13.704802 5085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "3370421101d54dd5b0110b8e24b2b80b"
format_stamp: "Formatted at 2026-09-25 21:56:13 on dist-test-slave-2g0f"
server_key: "58aaa65ca39bcc8d61b5220541f78d82"
server_key_iv: "d1c3a9226751b55349c7172a89b5c30d"
server_key_version: "encryptionkey@0"
I20260925 21:56:13.704865 5085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:13.713624 5085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:13.715759 5085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:13.715924 5085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:13.716161 5085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:13.716446 5085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:13.716478 5085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:13.716500 5085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:13.716511 5085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:13.721765 5085 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:34253
I20260925 21:56:13.721807 5213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:34253 every 8 connection(s)
I20260925 21:56:13.722105 5085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
Sep 25 21:56:13 dist-test-slave-2g0f krb5kdc[4708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373373, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:13.729713 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 5085
I20260925 21:56:13.729810 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:13.729997 2331 external_mini_cluster.cc:1428] Setting key 72808c7689b1e6a74b9f082f6bdda7a8
I20260925 21:56:13.731087 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:13.722769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:44303 (local address 127.2.70.254:39897)
0925 21:56:13.722878 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:13.722879 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:13.723043 (+ 164us) server_negotiation.cc:407] Connection header received
0925 21:56:13.723538 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:13.723539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:13.723551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:13.723619 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:13.724572 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:13.725079 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:13.726020 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:13.726230 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:13.726334 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:13.728380 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:13.728382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:13.728382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:13.728389 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:13.729754 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:13.730164 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:13.730165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:13.730165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:13.730215 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:13.730452 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:13.730453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:13.730453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:13.730561 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:13.730618 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:13.730893 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 21:56:13.730991 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20260925 21:56:13.731256 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:13.722748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:39897 (local address 127.2.70.195:44303)
0925 21:56:13.722924 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:13.722965 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:13.723439 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:13.723674 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:13.723678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:13.724283 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:13.724481 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:13.724486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:13.725151 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:13.725152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:13.725947 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:13.725950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:13.726346 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:13.726347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:13.726442 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:13.726871 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:13.726879 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:13.728282 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:13.729836 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:13.729843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:13.729845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:13.730090 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:13.730267 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:13.730268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:13.730268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:13.730386 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:13.730619 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:13.730626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:13.730766 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 21:56:13.730862 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 21:56:13.731034 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260925 21:56:13.731664 5214 heartbeater.cc:347] Connected to a master server at 127.2.70.254:39897
I20260925 21:56:13.731750 5214 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:13.731952 5214 heartbeater.cc:511] Master 127.2.70.254:39897 requested a full tablet report, sending...
I20260925 21:56:13.732271 4753 ts_manager.cc:194] Registered new tserver with Master: 3370421101d54dd5b0110b8e24b2b80b (127.2.70.195:34253)
I20260925 21:56:13.732614 4753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:44303
I20260925 21:56:13.741534 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 21:56:13.752444 5216 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:13.751396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.2.70.254:39897)
0925 21:56:13.751475 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:13.751477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:13.751621 (+ 144us) server_negotiation.cc:407] Connection header received
0925 21:56:13.751655 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:13.751656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:13.751666 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:13.751712 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:13.752384 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38492: BlockingRecv error: recv got EOF from 127.0.0.1:38492 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 21:56:13.753445 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 4815
I20260925 21:56:13.757339 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 4950
I20260925 21:56:13.761390 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 5085
I20260925 21:56:13.765501 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 4721
2026-09-25T21:56:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5029 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:13 dist-test-slave-2g0f krb5kdc[5227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:13 dist-test-slave-2g0f krb5kdc[5227](info): set up 2 sockets
Sep 25 21:56:13 dist-test-slave-2g0f krb5kdc[5227](info): commencing operation
krb5kdc: starting...
W20260925 21:56:14.727586 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:14 dist-test-slave-2g0f krb5kdc[5227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:56:14.847069 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:38425
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:38425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:14.929301 5240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:14.929512 5240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:14.929533 5240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:14.930644 5240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:14.930680 5240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:14.930691 5240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:14.930703 5240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:14.930722 5240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:14.932267 5240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:38425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:38425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:14.932476 5240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:14.932662 5240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:14.937273 5245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:14.937394 5246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:14.937443 5248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:14.937561 5240 server_base.cc:1034] running on GCE node
I20260925 21:56:14.937808 5240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:14.938082 5240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:14.939229 5240 hybrid_clock.cc:648] HybridClock initialized: now 1790373374939210 us; error 30 us; skew 500 ppm
Sep 25 21:56:14 dist-test-slave-2g0f krb5kdc[5227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373374, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:14.947239 5240 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:56:14.947705 5240 webserver.cc:460] Webserver started at http://127.2.70.254:44271/ using document root <none> and password file <none>
I20260925 21:56:14.947983 5240 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:14.948038 5240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:14.948166 5240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:14.949239 5240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "5fe6f6274f1d4bdc936e110fcd8c1892"
format_stamp: "Formatted at 2026-09-25 21:56:14 on dist-test-slave-2g0f"
server_key: "7a6a67e18cf4d6687479b407d447cc62"
server_key_iv: "7f0c7db3c5fe1e6dbe569503df50edf5"
server_key_version: "encryptionkey@0"
I20260925 21:56:14.949566 5240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "5fe6f6274f1d4bdc936e110fcd8c1892"
format_stamp: "Formatted at 2026-09-25 21:56:14 on dist-test-slave-2g0f"
server_key: "7a6a67e18cf4d6687479b407d447cc62"
server_key_iv: "7f0c7db3c5fe1e6dbe569503df50edf5"
server_key_version: "encryptionkey@0"
I20260925 21:56:14.950918 5240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:14.951716 5255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:14.951931 5240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:14.952023 5240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "5fe6f6274f1d4bdc936e110fcd8c1892"
format_stamp: "Formatted at 2026-09-25 21:56:14 on dist-test-slave-2g0f"
server_key: "7a6a67e18cf4d6687479b407d447cc62"
server_key_iv: "7f0c7db3c5fe1e6dbe569503df50edf5"
server_key_version: "encryptionkey@0"
I20260925 21:56:14.952097 5240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:14.959970 5240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:14.962198 5240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:14.962343 5240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:14.966298 5240 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:38425
I20260925 21:56:14.966378 5307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:38425 every 8 connection(s)
I20260925 21:56:14.966694 5240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:14.967581 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:14.970036 5308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892: Bootstrap starting.
I20260925 21:56:14.970651 5308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:14.970955 5308 log.cc:824] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:14.971822 5308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892: No bootstrap required, opened a new log
I20260925 21:56:14.971864 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 5240
I20260925 21:56:14.971946 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:14.972112 2331 external_mini_cluster.cc:1428] Setting key 50404dcba6defc425e539e2dfe6de648
I20260925 21:56:14.974910 5308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe6f6274f1d4bdc936e110fcd8c1892" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38425 } }
I20260925 21:56:14.975075 5308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:14.975107 5308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe6f6274f1d4bdc936e110fcd8c1892, State: Initialized, Role: FOLLOWER
I20260925 21:56:14.975243 5308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe6f6274f1d4bdc936e110fcd8c1892" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38425 } }
I20260925 21:56:14.975324 5308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:14.975358 5308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:14.975405 5308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:14.976099 5308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe6f6274f1d4bdc936e110fcd8c1892" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38425 } }
I20260925 21:56:14.976213 5308 leader_election.cc:302] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fe6f6274f1d4bdc936e110fcd8c1892; no voters:
I20260925 21:56:14.976387 5308 leader_election.cc:288] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:14.976435 5313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:14.976569 5313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [term 1 LEADER]: Becoming Leader. State: Replica: 5fe6f6274f1d4bdc936e110fcd8c1892, State: Running, Role: LEADER
I20260925 21:56:14.976714 5308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:14.976709 5313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe6f6274f1d4bdc936e110fcd8c1892" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38425 } }
Sep 25 21:56:14 dist-test-slave-2g0f krb5kdc[5227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:14.978936 5314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe6f6274f1d4bdc936e110fcd8c1892. Latest consensus state: current_term: 1 leader_uuid: "5fe6f6274f1d4bdc936e110fcd8c1892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe6f6274f1d4bdc936e110fcd8c1892" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38425 } } }
I20260925 21:56:14.979099 5315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fe6f6274f1d4bdc936e110fcd8c1892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe6f6274f1d4bdc936e110fcd8c1892" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38425 } } }
W20260925 21:56:14.979406 5327 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 21:56:14.979718 5314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:14.979730 5315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892 [sys.catalog]: This master's current role is: LEADER
W20260925 21:56:14.979743 5327 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:56:14.980154 5330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:14.980342 5330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:14.983518 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:14.972697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.2.70.254:38425)
0925 21:56:14.972882 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:14.972889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:14.972916 (+ 27us) server_negotiation.cc:407] Connection header received
0925 21:56:14.973333 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:14.973340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:14.973393 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:14.973515 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:14.974376 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:14.975378 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:14.976214 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:14.976862 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:14.976971 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:14.980056 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:14.980061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:14.980061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:14.980075 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:14.981877 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:14.982244 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:14.982248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:14.982248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:14.982339 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:14.982526 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:14.982526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:14.982527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:14.982782 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:14.982936 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:14.983190 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 21:56:14.983341 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260925 21:56:14.984589 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 21:56:14.985442 5330 catalog_manager.cc:1347] Generated new cluster ID: 14f0121864514f87b92ef23b6d765c08
I20260925 21:56:14.985504 5330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:14.999306 5330 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:15.000031 5330 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:15.008087 5330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fe6f6274f1d4bdc936e110fcd8c1892: Generated new TSK 0
I20260925 21:56:15.008364 5330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:56:15.012274 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:15.008173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.2.70.254:38425)
0925 21:56:15.008217 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:15.008219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:15.008344 (+ 125us) server_negotiation.cc:407] Connection header received
0925 21:56:15.008458 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:15.008459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:15.008470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:15.008499 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:15.008961 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:15.009533 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:15.010287 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:15.010511 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:15.010568 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:15.011526 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:15.011528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:15.011529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:15.011814 (+ 285us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:15.011899 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:15.011909 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:15.011965 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:15.012116 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 21:56:15.012172 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 21:56:15.015802 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 5240
2026-09-25T21:56:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1248 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:15 dist-test-slave-2g0f krb5kdc[5341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:15 dist-test-slave-2g0f krb5kdc[5341](info): set up 2 sockets
Sep 25 21:56:15 dist-test-slave-2g0f krb5kdc[5341](info): commencing operation
krb5kdc: starting...
W20260925 21:56:15.979547 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:56:16.094348 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:37019
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:37019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:16.174434 5354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:16.174641 5354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:16.174661 5354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:16.175755 5354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:16.175787 5354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:16.175799 5354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:16.175810 5354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:16.175822 5354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:16.177263 5354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:37019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:37019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:16.177448 5354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:16.177618 5354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:16.182183 5360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:16.182183 5362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:16.182241 5359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:16.182340 5354 server_base.cc:1034] running on GCE node
I20260925 21:56:16.182663 5354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:16.182942 5354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:16.184113 5354 hybrid_clock.cc:648] HybridClock initialized: now 1790373376184090 us; error 29 us; skew 500 ppm
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:16.186698 5354 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:56:16.187112 5354 webserver.cc:460] Webserver started at http://127.2.70.254:46517/ using document root <none> and password file <none>
I20260925 21:56:16.187330 5354 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:16.187378 5354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:16.187505 5354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:16.188648 5354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "669da1e1300246da8231bdb7cfb0a90c"
format_stamp: "Formatted at 2026-09-25 21:56:16 on dist-test-slave-2g0f"
server_key: "cc1fb9f9ba0597935416c3cb63fbcdd6"
server_key_iv: "2740a6c839877b5d0acd2aa2d9c18e39"
server_key_version: "encryptionkey@0"
I20260925 21:56:16.188995 5354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "669da1e1300246da8231bdb7cfb0a90c"
format_stamp: "Formatted at 2026-09-25 21:56:16 on dist-test-slave-2g0f"
server_key: "cc1fb9f9ba0597935416c3cb63fbcdd6"
server_key_iv: "2740a6c839877b5d0acd2aa2d9c18e39"
server_key_version: "encryptionkey@0"
I20260925 21:56:16.190240 5354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:16.190958 5369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:16.191126 5354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:16.191200 5354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "669da1e1300246da8231bdb7cfb0a90c"
format_stamp: "Formatted at 2026-09-25 21:56:16 on dist-test-slave-2g0f"
server_key: "cc1fb9f9ba0597935416c3cb63fbcdd6"
server_key_iv: "2740a6c839877b5d0acd2aa2d9c18e39"
server_key_version: "encryptionkey@0"
I20260925 21:56:16.191327 5354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:16.206811 5354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:16.208811 5354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:16.208948 5354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:16.212694 5354 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:37019
I20260925 21:56:16.212734 5421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:37019 every 8 connection(s)
I20260925 21:56:16.213057 5354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:16.214082 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:16.216509 5422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c: Bootstrap starting.
I20260925 21:56:16.217034 5422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:16.217278 5422 log.cc:824] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:16.217981 5422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c: No bootstrap required, opened a new log
I20260925 21:56:16.219535 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 5354
I20260925 21:56:16.219628 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:16.219789 2331 external_mini_cluster.cc:1428] Setting key e63593d3902fbdb97e3ce9e149d1e7fc
I20260925 21:56:16.219666 5422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669da1e1300246da8231bdb7cfb0a90c" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 37019 } }
I20260925 21:56:16.219820 5422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:16.219920 5422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669da1e1300246da8231bdb7cfb0a90c, State: Initialized, Role: FOLLOWER
I20260925 21:56:16.220086 5422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669da1e1300246da8231bdb7cfb0a90c" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 37019 } }
I20260925 21:56:16.220193 5422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:16.220283 5422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:16.220345 5422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:16.220983 5422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669da1e1300246da8231bdb7cfb0a90c" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 37019 } }
I20260925 21:56:16.221088 5422 leader_election.cc:302] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 669da1e1300246da8231bdb7cfb0a90c; no voters:
I20260925 21:56:16.221276 5422 leader_election.cc:288] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:16.221361 5427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:16.221532 5427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [term 1 LEADER]: Becoming Leader. State: Replica: 669da1e1300246da8231bdb7cfb0a90c, State: Running, Role: LEADER
I20260925 21:56:16.221551 5422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:16.221664 5427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669da1e1300246da8231bdb7cfb0a90c" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 37019 } }
I20260925 21:56:16.222002 5428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "669da1e1300246da8231bdb7cfb0a90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669da1e1300246da8231bdb7cfb0a90c" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 37019 } } }
I20260925 21:56:16.222077 5428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:16.222261 5429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 669da1e1300246da8231bdb7cfb0a90c. Latest consensus state: current_term: 1 leader_uuid: "669da1e1300246da8231bdb7cfb0a90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669da1e1300246da8231bdb7cfb0a90c" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 37019 } } }
I20260925 21:56:16.222648 5432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:16.223147 5432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:16.223886 5429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c [sys.catalog]: This master's current role is: LEADER
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:16.228314 5432 catalog_manager.cc:1347] Generated new cluster ID: 4eab33b9267941de8cac12b5ff31d895
I20260925 21:56:16.228376 5432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:16.230973 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:16.220435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.2.70.254:37019)
0925 21:56:16.220593 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:16.220601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:16.220626 (+ 25us) server_negotiation.cc:407] Connection header received
0925 21:56:16.220934 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:16.220940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:16.220986 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:16.221113 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:16.221943 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.223353 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.224452 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.224856 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.224947 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:16.227094 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:16.227099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:16.227099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:16.227111 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:16.229360 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:16.229763 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:16.229766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:16.229766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:16.229879 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:16.230079 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:16.230080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:16.230080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:16.230367 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:16.230499 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:16.230624 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 21:56:16.230748 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260925 21:56:16.256484 5432 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:16.257129 5432 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:16.267182 5432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 669da1e1300246da8231bdb7cfb0a90c: Generated new TSK 0
I20260925 21:56:16.267378 5432 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:56:16.294106 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:37019
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:16.374907 5448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:16.375100 5448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:16.375123 5448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:16.376255 5448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:16.376288 5448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:16.376313 5448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:16.377672 5448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:37019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:16.377851 5448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:16.378026 5448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:16.382835 5453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:16.382844 5454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:16.382907 5456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:16.383117 5448 server_base.cc:1034] running on GCE node
I20260925 21:56:16.383442 5448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:16.383668 5448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:16.384804 5448 hybrid_clock.cc:648] HybridClock initialized: now 1790373376384784 us; error 37 us; skew 500 ppm
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:16.387133 5448 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:56:16.387568 5448 webserver.cc:460] Webserver started at http://127.2.70.193:43121/ using document root <none> and password file <none>
I20260925 21:56:16.387753 5448 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:16.387795 5448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:16.387920 5448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:16.388921 5448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "9d7eb09e5ac74c82a01425a5409a395c"
format_stamp: "Formatted at 2026-09-25 21:56:16 on dist-test-slave-2g0f"
server_key: "6e0547d7de4e4cf811584d3b7842b26a"
server_key_iv: "3bcb6c5f0c7de4cbc70df6c17321e6c2"
server_key_version: "encryptionkey@0"
I20260925 21:56:16.389204 5448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "9d7eb09e5ac74c82a01425a5409a395c"
format_stamp: "Formatted at 2026-09-25 21:56:16 on dist-test-slave-2g0f"
server_key: "6e0547d7de4e4cf811584d3b7842b26a"
server_key_iv: "3bcb6c5f0c7de4cbc70df6c17321e6c2"
server_key_version: "encryptionkey@0"
I20260925 21:56:16.390385 5448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:16.391034 5463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:16.391227 5448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:16.391295 5448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "9d7eb09e5ac74c82a01425a5409a395c"
format_stamp: "Formatted at 2026-09-25 21:56:16 on dist-test-slave-2g0f"
server_key: "6e0547d7de4e4cf811584d3b7842b26a"
server_key_iv: "3bcb6c5f0c7de4cbc70df6c17321e6c2"
server_key_version: "encryptionkey@0"
I20260925 21:56:16.391347 5448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:16.406083 5448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:16.408257 5448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:16.408370 5448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:16.408560 5448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:16.408885 5448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:16.408917 5448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:16.408948 5448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:16.408972 5448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:16.414217 5448 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:40259
I20260925 21:56:16.414284 5576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:40259 every 8 connection(s)
I20260925 21:56:16.414572 5448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:16.418819 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 5448
I20260925 21:56:16.418931 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:16.419145 2331 external_mini_cluster.cc:1428] Setting key 442f6dfdf46466d23b72671152689840
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:16.424401 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:16.415277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:48487 (local address 127.2.70.254:37019)
0925 21:56:16.415343 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:16.415345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:16.415535 (+ 190us) server_negotiation.cc:407] Connection header received
0925 21:56:16.416103 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:16.416104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:16.416122 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:16.416168 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:16.417125 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.417633 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.418651 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.418861 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.418921 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:16.421372 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:16.421373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:16.421373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:16.421386 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:16.422903 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:16.423341 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:16.423342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:16.423343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:16.423408 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:16.423718 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:16.423719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:16.423720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:16.423888 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:16.423961 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:16.424201 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 21:56:16.424301 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 21:56:16.424465 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:16.415238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:37019 (local address 127.2.70.193:48487)
0925 21:56:16.415413 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:16.415452 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:16.416006 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:16.416256 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:16.416260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:16.416835 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:16.417029 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:16.417034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.417720 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.417721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:16.418545 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:16.418548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.418969 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.418970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:16.419156 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:16.419679 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:16.419688 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:16.421234 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:16.422990 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:16.422996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:16.422998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:16.423258 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:16.423493 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:16.423494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:16.423494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:16.423644 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:16.423954 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:16.423957 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:16.424078 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 21:56:16.424160 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 21:56:16.424298 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260925 21:56:16.425258 5577 heartbeater.cc:347] Connected to a master server at 127.2.70.254:37019
I20260925 21:56:16.425346 5577 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:16.425519 5577 heartbeater.cc:511] Master 127.2.70.254:37019 requested a full tablet report, sending...
I20260925 21:56:16.426062 5386 ts_manager.cc:194] Registered new tserver with Master: 9d7eb09e5ac74c82a01425a5409a395c (127.2.70.193:40259)
I20260925 21:56:16.426982 5386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:48487
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:56:16.445302 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:37019
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:16.523000 5582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:16.523204 5582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:16.523226 5582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:16.524405 5582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:16.524446 5582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:16.524471 5582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:16.526227 5582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:37019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:16.526458 5582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:16.526659 5582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:16.531387 5590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:16.531387 5587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:16.531565 5582 server_base.cc:1034] running on GCE node
W20260925 21:56:16.531418 5588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:16.531817 5582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:16.532073 5582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:16.533242 5582 hybrid_clock.cc:648] HybridClock initialized: now 1790373376533226 us; error 29 us; skew 500 ppm
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:16.535713 5582 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:56:16.536258 5582 webserver.cc:460] Webserver started at http://127.2.70.194:45061/ using document root <none> and password file <none>
I20260925 21:56:16.536468 5582 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:16.536514 5582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:16.536638 5582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:16.537669 5582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "9d4cf91ae81d424bb4a2571a8a67268e"
format_stamp: "Formatted at 2026-09-25 21:56:16 on dist-test-slave-2g0f"
server_key: "81c20f72a02f3a17b1c9c6f091e7ce99"
server_key_iv: "8de2e5f95f2160f957c6cbb03fb8a528"
server_key_version: "encryptionkey@0"
I20260925 21:56:16.537987 5582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "9d4cf91ae81d424bb4a2571a8a67268e"
format_stamp: "Formatted at 2026-09-25 21:56:16 on dist-test-slave-2g0f"
server_key: "81c20f72a02f3a17b1c9c6f091e7ce99"
server_key_iv: "8de2e5f95f2160f957c6cbb03fb8a528"
server_key_version: "encryptionkey@0"
I20260925 21:56:16.539273 5582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:16.540053 5597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:16.540201 5582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:16.540307 5582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "9d4cf91ae81d424bb4a2571a8a67268e"
format_stamp: "Formatted at 2026-09-25 21:56:16 on dist-test-slave-2g0f"
server_key: "81c20f72a02f3a17b1c9c6f091e7ce99"
server_key_iv: "8de2e5f95f2160f957c6cbb03fb8a528"
server_key_version: "encryptionkey@0"
I20260925 21:56:16.540372 5582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:16.551107 5582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:16.553258 5582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:16.553393 5582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:16.553602 5582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:16.553932 5582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:16.553966 5582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:16.553995 5582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:16.554015 5582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:16.559379 5582 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:44863
I20260925 21:56:16.559445 5710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:44863 every 8 connection(s)
I20260925 21:56:16.559719 5582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:16.559948 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 5582
I20260925 21:56:16.560058 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:16.560230 2331 external_mini_cluster.cc:1428] Setting key abe825588a05103d9be3ecdabbcde4b3
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
I20260925 21:56:16.569617 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:16.560490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:58871 (local address 127.2.70.254:37019)
0925 21:56:16.560543 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:16.560545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:16.560799 (+ 254us) server_negotiation.cc:407] Connection header received
0925 21:56:16.561338 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:16.561339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:16.561352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:16.561431 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:16.562478 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.562992 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.564413 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.564603 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.564655 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:16.566993 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:16.566994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:16.566994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:16.567001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:16.568328 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:16.568733 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:16.568734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:16.568734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:16.568795 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:16.569005 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:16.569005 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:16.569006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:16.569129 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:16.569180 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:16.569467 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 21:56:16.569549 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 21:56:16.569689 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:16.560497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:37019 (local address 127.2.70.194:58871)
0925 21:56:16.560658 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:16.560704 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:16.561234 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:16.561485 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:16.561489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:16.562089 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:16.562381 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:16.562387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.563079 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.563080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:16.564332 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:16.564335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.564679 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.564680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:16.564820 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:16.565310 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:16.565320 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:16.566902 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:16.568408 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:16.568414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:16.568415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:16.568666 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:16.568847 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:16.568847 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:16.568848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:16.568946 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:16.569182 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:16.569186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:16.569314 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 21:56:16.569399 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 21:56:16.569521 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260925 21:56:16.570272 5711 heartbeater.cc:347] Connected to a master server at 127.2.70.254:37019
I20260925 21:56:16.570369 5711 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:16.570523 5711 heartbeater.cc:511] Master 127.2.70.254:37019 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
I20260925 21:56:16.570941 5386 ts_manager.cc:194] Registered new tserver with Master: 9d4cf91ae81d424bb4a2571a8a67268e (127.2.70.194:44863)
I20260925 21:56:16.571408 5386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:58871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:56:16.586527 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:37019
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:16.666340 5716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:16.666559 5716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:16.666594 5716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:16.667755 5716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:16.667796 5716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:16.667860 5716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:16.669315 5716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:37019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:16.669544 5716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:16.669742 5716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:16.674438 5722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:16.674628 5716 server_base.cc:1034] running on GCE node
W20260925 21:56:16.674589 5721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:16.674589 5724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:16.674934 5716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:16.675164 5716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:16.676287 5716 hybrid_clock.cc:648] HybridClock initialized: now 1790373376676275 us; error 27 us; skew 500 ppm
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:16.678516 5716 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:56:16.678907 5716 webserver.cc:460] Webserver started at http://127.2.70.195:43757/ using document root <none> and password file <none>
I20260925 21:56:16.679077 5716 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:16.679114 5716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:16.679199 5716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:16.680222 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "c06ae26821674e3a95911483667a79ce"
format_stamp: "Formatted at 2026-09-25 21:56:16 on dist-test-slave-2g0f"
server_key: "957cbc285dd0a58fb9fdc4a7a7e1e845"
server_key_iv: "6d6a034ec96c72868006230d4ca269eb"
server_key_version: "encryptionkey@0"
I20260925 21:56:16.680505 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "c06ae26821674e3a95911483667a79ce"
format_stamp: "Formatted at 2026-09-25 21:56:16 on dist-test-slave-2g0f"
server_key: "957cbc285dd0a58fb9fdc4a7a7e1e845"
server_key_iv: "6d6a034ec96c72868006230d4ca269eb"
server_key_version: "encryptionkey@0"
I20260925 21:56:16.681654 5716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:16.682339 5731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:16.682482 5716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:16.682556 5716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "c06ae26821674e3a95911483667a79ce"
format_stamp: "Formatted at 2026-09-25 21:56:16 on dist-test-slave-2g0f"
server_key: "957cbc285dd0a58fb9fdc4a7a7e1e845"
server_key_iv: "6d6a034ec96c72868006230d4ca269eb"
server_key_version: "encryptionkey@0"
I20260925 21:56:16.682621 5716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:16.709618 5716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:16.711671 5716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:16.711790 5716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:16.712013 5716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:16.712311 5716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:16.712342 5716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:16.712364 5716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:16.712387 5716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:16.717792 5716 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:44641
I20260925 21:56:16.717830 5844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:44641 every 8 connection(s)
I20260925 21:56:16.718130 5716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:16.721472 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 5716
I20260925 21:56:16.721568 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:16.721756 2331 external_mini_cluster.cc:1428] Setting key bf56960277fa8fa593d7ee8d8dcbc26f
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:16.727295 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:16.718867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:56735 (local address 127.2.70.254:37019)
0925 21:56:16.718930 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:16.718932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:16.719110 (+ 178us) server_negotiation.cc:407] Connection header received
0925 21:56:16.719628 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:16.719629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:16.719640 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:16.719693 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:16.720708 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.721208 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.722551 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.722865 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.722933 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:16.724950 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:16.724952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:16.724952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:16.724959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:16.726182 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:16.726541 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:16.726542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:16.726543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:16.726595 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:16.726784 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:16.726784 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:16.726784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:16.726878 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:16.726922 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:16.727156 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 21:56:16.727217 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 21:56:16.727530 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:16.718826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:37019 (local address 127.2.70.195:56735)
0925 21:56:16.718993 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:16.719031 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:16.719533 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:16.719770 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:16.719773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:16.720413 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:16.720611 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:16.720616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.721286 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.721287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:16.722477 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:16.722480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.722939 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.722940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:16.723044 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:16.723534 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:16.723543 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:16.724871 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:16.726263 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:16.726268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:16.726270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:16.726478 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:16.726643 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:16.726644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:16.726644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:16.726738 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:16.726936 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:16.726941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:16.727054 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 21:56:16.727134 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 21:56:16.727313 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260925 21:56:16.727986 5845 heartbeater.cc:347] Connected to a master server at 127.2.70.254:37019
I20260925 21:56:16.728072 5845 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:16.728295 5845 heartbeater.cc:511] Master 127.2.70.254:37019 requested a full tablet report, sending...
I20260925 21:56:16.728642 5386 ts_manager.cc:194] Registered new tserver with Master: c06ae26821674e3a95911483667a79ce (127.2.70.195:44641)
I20260925 21:56:16.728984 5386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:56735
I20260925 21:56:16.733439 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373376, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373376, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:16.755183 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:16.748169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.2.70.254:37019)
0925 21:56:16.748228 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:16.748231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:16.748365 (+ 134us) server_negotiation.cc:407] Connection header received
0925 21:56:16.748443 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:16.748444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:16.748456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:16.748503 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:16.749054 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.749549 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.750334 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:16.750521 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:16.750577 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:16.752799 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:16.752801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:16.752801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:16.752810 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:16.754071 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:16.754431 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:16.754432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:16.754433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:16.754485 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:16.754641 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:16.754642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:16.754642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:16.754799 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:16.754888 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:16.755010 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 21:56:16.755088 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 21:56:16.755414 5386 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54386
I20260925 21:56:16.755795 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 5448
I20260925 21:56:16.760334 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 5582
I20260925 21:56:16.764230 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 5716
I20260925 21:56:16.767627 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 5354
2026-09-25T21:56:16Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1752 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5857](info): set up 2 sockets
Sep 25 21:56:16 dist-test-slave-2g0f krb5kdc[5857](info): commencing operation
krb5kdc: starting...
W20260925 21:56:17.723551 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:17 dist-test-slave-2g0f krb5kdc[5857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:56:17.840790 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:46253
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:46253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:17.920431 5870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:17.920655 5870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:17.920680 5870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:17.922314 5870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:17.922375 5870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:17.922389 5870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:17.922401 5870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:17.922412 5870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:17.924163 5870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:46253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:46253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:17.924415 5870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:17.924649 5870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:17.929384 5875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:17.929388 5878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:17.929503 5876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:17.929425 5870 server_base.cc:1034] running on GCE node
I20260925 21:56:17.929883 5870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:17.930171 5870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:17.931329 5870 hybrid_clock.cc:648] HybridClock initialized: now 1790373377931329 us; error 55 us; skew 500 ppm
Sep 25 21:56:17 dist-test-slave-2g0f krb5kdc[5857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373377, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:17.933849 5870 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:56:17.934314 5870 webserver.cc:460] Webserver started at http://127.2.70.254:34755/ using document root <none> and password file <none>
I20260925 21:56:17.934533 5870 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:17.934587 5870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:17.934716 5870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:17.935809 5870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "64efaffab60647b8a55b897b897dffaa"
format_stamp: "Formatted at 2026-09-25 21:56:17 on dist-test-slave-2g0f"
server_key: "7560485f68973dbfe41e3fe2a012e2df"
server_key_iv: "0043cfd8321ddc964c64341fbe34569a"
server_key_version: "encryptionkey@0"
I20260925 21:56:17.936193 5870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "64efaffab60647b8a55b897b897dffaa"
format_stamp: "Formatted at 2026-09-25 21:56:17 on dist-test-slave-2g0f"
server_key: "7560485f68973dbfe41e3fe2a012e2df"
server_key_iv: "0043cfd8321ddc964c64341fbe34569a"
server_key_version: "encryptionkey@0"
I20260925 21:56:17.937549 5870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:17.938318 5885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:17.938474 5870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:17.938568 5870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "64efaffab60647b8a55b897b897dffaa"
format_stamp: "Formatted at 2026-09-25 21:56:17 on dist-test-slave-2g0f"
server_key: "7560485f68973dbfe41e3fe2a012e2df"
server_key_iv: "0043cfd8321ddc964c64341fbe34569a"
server_key_version: "encryptionkey@0"
I20260925 21:56:17.938632 5870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:17.948159 5870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:17.949955 5870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:17.950088 5870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:17.953856 5870 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:46253
I20260925 21:56:17.953941 5937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:46253 every 8 connection(s)
I20260925 21:56:17.954352 5870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:17.955011 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 5870
I20260925 21:56:17.955040 5938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:17.955113 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:17.955299 2331 external_mini_cluster.cc:1428] Setting key 5f4a627542bd1795ce3415c88a38c8f5
I20260925 21:56:17.958106 5938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa: Bootstrap starting.
I20260925 21:56:17.958684 5938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:17.958925 5938 log.cc:824] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:17.959508 5938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa: No bootstrap required, opened a new log
Sep 25 21:56:17 dist-test-slave-2g0f krb5kdc[5857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:17.961583 5938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64efaffab60647b8a55b897b897dffaa" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46253 } }
I20260925 21:56:17.961742 5938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:17.961768 5938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64efaffab60647b8a55b897b897dffaa, State: Initialized, Role: FOLLOWER
I20260925 21:56:17.961891 5938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64efaffab60647b8a55b897b897dffaa" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46253 } }
I20260925 21:56:17.961952 5938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:17.961985 5938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:17.962028 5938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:17.962556 5938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64efaffab60647b8a55b897b897dffaa" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46253 } }
I20260925 21:56:17.962662 5938 leader_election.cc:302] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64efaffab60647b8a55b897b897dffaa; no voters:
I20260925 21:56:17.962841 5938 leader_election.cc:288] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:17.962888 5943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:17.963020 5943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [term 1 LEADER]: Becoming Leader. State: Replica: 64efaffab60647b8a55b897b897dffaa, State: Running, Role: LEADER
I20260925 21:56:17.963101 5938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:17.963117 5943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64efaffab60647b8a55b897b897dffaa" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46253 } }
I20260925 21:56:17.963434 5943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64efaffab60647b8a55b897b897dffaa. Latest consensus state: current_term: 1 leader_uuid: "64efaffab60647b8a55b897b897dffaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64efaffab60647b8a55b897b897dffaa" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46253 } } }
I20260925 21:56:17.963577 5943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:17.964285 5945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64efaffab60647b8a55b897b897dffaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64efaffab60647b8a55b897b897dffaa" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46253 } } }
I20260925 21:56:17.964390 5945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa [sys.catalog]: This master's current role is: LEADER
W20260925 21:56:17.964879 5959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:56:17.964929 5959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:56:17.965135 5950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:17.965319 5950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:17.965812 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:17.955800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.2.70.254:46253)
0925 21:56:17.955980 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:17.955987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:17.956013 (+ 26us) server_negotiation.cc:407] Connection header received
0925 21:56:17.956543 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:17.956551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:17.956600 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:17.956732 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:17.957401 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:17.958328 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:17.959115 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:17.959530 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:17.959633 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:17.962234 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:17.962237 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:17.962237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:17.962247 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:17.964197 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:17.964578 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:17.964581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:17.964581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:17.964697 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:17.964919 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:17.964920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:17.964921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:17.965235 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:17.965400 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:17.965477 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 21:56:17.965607 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260925 21:56:17.966745 5950 catalog_manager.cc:1347] Generated new cluster ID: 1c6c25ad70574af08eb48002a22f613e
I20260925 21:56:17.966784 5950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:17.975518 5950 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:17.976118 5950 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:17.984417 5950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64efaffab60647b8a55b897b897dffaa: Generated new TSK 0
I20260925 21:56:17.984617 5950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:56:18.015288 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46253
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:18.096280 5964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:18.096493 5964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:18.096514 5964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:18.097653 5964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:18.097692 5964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:18.097718 5964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:18.099200 5964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:18.099408 5964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:18.099609 5964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:18.104316 5969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:18.104331 5970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:18.104313 5972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:18.104524 5964 server_base.cc:1034] running on GCE node
I20260925 21:56:18.104740 5964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:18.104978 5964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:18.106123 5964 hybrid_clock.cc:648] HybridClock initialized: now 1790373378106109 us; error 24 us; skew 500 ppm
Sep 25 21:56:18 dist-test-slave-2g0f krb5kdc[5857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:18.108522 5964 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:56:18.108952 5964 webserver.cc:460] Webserver started at http://127.2.70.193:42719/ using document root <none> and password file <none>
I20260925 21:56:18.109153 5964 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:18.109196 5964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:18.109318 5964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:18.110389 5964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "323d640238a64003af61f8d0b95673d3"
format_stamp: "Formatted at 2026-09-25 21:56:18 on dist-test-slave-2g0f"
server_key: "f80ebf5dffacfd783de11a6a6f0d0a71"
server_key_iv: "42593f4e95bdb4d6c5a8d569a2115478"
server_key_version: "encryptionkey@0"
I20260925 21:56:18.110713 5964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "323d640238a64003af61f8d0b95673d3"
format_stamp: "Formatted at 2026-09-25 21:56:18 on dist-test-slave-2g0f"
server_key: "f80ebf5dffacfd783de11a6a6f0d0a71"
server_key_iv: "42593f4e95bdb4d6c5a8d569a2115478"
server_key_version: "encryptionkey@0"
I20260925 21:56:18.112005 5964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 21:56:18.112720 5979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:18.112862 5964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:18.112936 5964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "323d640238a64003af61f8d0b95673d3"
format_stamp: "Formatted at 2026-09-25 21:56:18 on dist-test-slave-2g0f"
server_key: "f80ebf5dffacfd783de11a6a6f0d0a71"
server_key_iv: "42593f4e95bdb4d6c5a8d569a2115478"
server_key_version: "encryptionkey@0"
I20260925 21:56:18.113000 5964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:18.129318 5964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:18.131074 5964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:18.131199 5964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:18.131429 5964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:18.131712 5964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:18.131743 5964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:18.131768 5964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:18.131793 5964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:18.137470 5964 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:40101
I20260925 21:56:18.137519 6092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:40101 every 8 connection(s)
I20260925 21:56:18.137840 5964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:18.140033 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 5964
I20260925 21:56:18.140120 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:18.140286 2331 external_mini_cluster.cc:1428] Setting key d2249577d586d75217cb30404527205b
Sep 25 21:56:18 dist-test-slave-2g0f krb5kdc[5857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:18.147743 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:18.138542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:33041 (local address 127.2.70.254:46253)
0925 21:56:18.138601 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:18.138603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:18.138810 (+ 207us) server_negotiation.cc:407] Connection header received
0925 21:56:18.139744 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:18.139745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:18.139756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:18.139819 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:18.140567 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.141219 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.142308 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.142610 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.142681 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:18.144931 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:18.144932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:18.144933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:18.144940 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:18.146451 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:18.146834 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:18.146835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:18.146835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:18.146895 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:18.147115 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:18.147116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:18.147116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:18.147255 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:18.147320 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:18.147549 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 21:56:18.147648 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 21:56:18.147800 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:18.138526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46253 (local address 127.2.70.193:33041)
0925 21:56:18.138686 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:18.138729 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:18.139650 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:18.139915 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:18.139919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:18.140217 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:18.140471 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:18.140478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.141378 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.141379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:18.142231 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:18.142234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.142688 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.142689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:18.142798 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:18.143277 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:18.143312 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:18.144837 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:18.146533 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:18.146540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:18.146542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:18.146765 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:18.146958 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:18.146959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:18.146959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:18.147058 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:18.147315 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:18.147318 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:18.147433 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 21:56:18.147508 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 21:56:18.147629 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
I20260925 21:56:18.148628 6093 heartbeater.cc:347] Connected to a master server at 127.2.70.254:46253
I20260925 21:56:18.148725 6093 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:18.148902 6093 heartbeater.cc:511] Master 127.2.70.254:46253 requested a full tablet report, sending...
I20260925 21:56:18.149407 5902 ts_manager.cc:194] Registered new tserver with Master: 323d640238a64003af61f8d0b95673d3 (127.2.70.193:40101)
I20260925 21:56:18.150012 5902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:33041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:56:18.166339 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46253
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:18.244552 6098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:18.244750 6098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:18.244772 6098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:18.245926 6098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:18.245966 6098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:18.245990 6098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:18.247455 6098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:18.247646 6098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:18.247852 6098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:18.252701 6103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:18.252712 6104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:18.252681 6106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:18.252786 6098 server_base.cc:1034] running on GCE node
I20260925 21:56:18.253089 6098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:18.253358 6098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:18.254510 6098 hybrid_clock.cc:648] HybridClock initialized: now 1790373378254482 us; error 45 us; skew 500 ppm
Sep 25 21:56:18 dist-test-slave-2g0f krb5kdc[5857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:18.257046 6098 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:56:18.257516 6098 webserver.cc:460] Webserver started at http://127.2.70.194:38303/ using document root <none> and password file <none>
I20260925 21:56:18.257755 6098 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:18.257807 6098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:18.257929 6098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:18.258946 6098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "6b205709e5054961a2c422f103d56d19"
format_stamp: "Formatted at 2026-09-25 21:56:18 on dist-test-slave-2g0f"
server_key: "311cd8c7df606a0187461e49d7b6a9f8"
server_key_iv: "c80342bb7f15b3f1ae2a04237b4cdaef"
server_key_version: "encryptionkey@0"
I20260925 21:56:18.259270 6098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "6b205709e5054961a2c422f103d56d19"
format_stamp: "Formatted at 2026-09-25 21:56:18 on dist-test-slave-2g0f"
server_key: "311cd8c7df606a0187461e49d7b6a9f8"
server_key_iv: "c80342bb7f15b3f1ae2a04237b4cdaef"
server_key_version: "encryptionkey@0"
I20260925 21:56:18.260601 6098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 21:56:18.261409 6113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:18.261574 6098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:18.261657 6098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "6b205709e5054961a2c422f103d56d19"
format_stamp: "Formatted at 2026-09-25 21:56:18 on dist-test-slave-2g0f"
server_key: "311cd8c7df606a0187461e49d7b6a9f8"
server_key_iv: "c80342bb7f15b3f1ae2a04237b4cdaef"
server_key_version: "encryptionkey@0"
I20260925 21:56:18.261716 6098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:18.270022 6098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:18.272380 6098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:18.272514 6098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:18.272740 6098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:18.273028 6098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:18.273061 6098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:18.273088 6098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:18.273113 6098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:18.278491 6098 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:33693
I20260925 21:56:18.278537 6226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:33693 every 8 connection(s)
I20260925 21:56:18.278843 6098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:18.280637 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 6098
I20260925 21:56:18.280750 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:18.280912 2331 external_mini_cluster.cc:1428] Setting key 1b36f2edf54a402bad6c3463fd9c83d2
Sep 25 21:56:18 dist-test-slave-2g0f krb5kdc[5857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:18.288316 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:18.279502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:47805 (local address 127.2.70.254:46253)
0925 21:56:18.279547 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:18.279549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:18.279738 (+ 189us) server_negotiation.cc:407] Connection header received
0925 21:56:18.280757 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:18.280760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:18.280772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:18.280829 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:18.281652 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.282143 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.283186 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.283387 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.283443 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:18.285619 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:18.285621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:18.285621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:18.285628 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:18.286961 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:18.287400 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:18.287401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:18.287401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:18.287449 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:18.287705 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:18.287706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:18.287706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:18.287803 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:18.287883 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:18.288183 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 21:56:18.288251 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 21:56:18.288573 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:18.279469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46253 (local address 127.2.70.194:47805)
0925 21:56:18.279620 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:18.279661 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:18.280648 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:18.280944 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:18.280948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:18.281289 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:18.281558 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:18.281565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.282244 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.282245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:18.283092 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:18.283095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.283457 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.283458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:18.283586 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:18.284115 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:18.284123 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:18.285493 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:18.287033 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:18.287040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:18.287042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:18.287331 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:18.287507 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:18.287508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:18.287508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:18.287637 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:18.287874 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:18.287880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:18.288056 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 21:56:18.288172 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 21:56:18.288342 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260925 21:56:18.289085 6227 heartbeater.cc:347] Connected to a master server at 127.2.70.254:46253
I20260925 21:56:18.289178 6227 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:18.289426 6227 heartbeater.cc:511] Master 127.2.70.254:46253 requested a full tablet report, sending...
I20260925 21:56:18.289779 5902 ts_manager.cc:194] Registered new tserver with Master: 6b205709e5054961a2c422f103d56d19 (127.2.70.194:33693)
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
I20260925 21:56:18.290150 5902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:47805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:56:18.306002 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46253
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:18.387111 6232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:18.387331 6232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:18.387352 6232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:18.388530 6232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:18.388569 6232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:18.388594 6232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:18.390040 6232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:18.390237 6232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:18.390405 6232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:18.395175 6238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:18.395316 6237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:18.395327 6232 server_base.cc:1034] running on GCE node
W20260925 21:56:18.395205 6240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:18.395670 6232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:18.395922 6232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:18.397078 6232 hybrid_clock.cc:648] HybridClock initialized: now 1790373378397066 us; error 26 us; skew 500 ppm
Sep 25 21:56:18 dist-test-slave-2g0f krb5kdc[5857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:18.399534 6232 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:56:18.400025 6232 webserver.cc:460] Webserver started at http://127.2.70.195:38917/ using document root <none> and password file <none>
I20260925 21:56:18.400288 6232 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:18.400343 6232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:18.400488 6232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:18.401546 6232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "52a075ece0c74721b9d38552c13fb171"
format_stamp: "Formatted at 2026-09-25 21:56:18 on dist-test-slave-2g0f"
server_key: "7860b523b3585f9f22848425ebde8ba8"
server_key_iv: "a61365b4a8e6845f14b9fbc62c016df0"
server_key_version: "encryptionkey@0"
I20260925 21:56:18.401914 6232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "52a075ece0c74721b9d38552c13fb171"
format_stamp: "Formatted at 2026-09-25 21:56:18 on dist-test-slave-2g0f"
server_key: "7860b523b3585f9f22848425ebde8ba8"
server_key_iv: "a61365b4a8e6845f14b9fbc62c016df0"
server_key_version: "encryptionkey@0"
I20260925 21:56:18.403236 6232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:18.403959 6247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:18.404103 6232 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:18.404192 6232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "52a075ece0c74721b9d38552c13fb171"
format_stamp: "Formatted at 2026-09-25 21:56:18 on dist-test-slave-2g0f"
server_key: "7860b523b3585f9f22848425ebde8ba8"
server_key_iv: "a61365b4a8e6845f14b9fbc62c016df0"
server_key_version: "encryptionkey@0"
I20260925 21:56:18.404263 6232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:18.420868 6232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:18.422897 6232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:18.423031 6232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:18.423261 6232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:18.423557 6232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:18.423590 6232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:18.423619 6232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:18.423636 6232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:18.428807 6232 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:37649
I20260925 21:56:18.428866 6360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:37649 every 8 connection(s)
I20260925 21:56:18.429167 6232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:18.430730 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 6232
I20260925 21:56:18.430821 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:18.430997 2331 external_mini_cluster.cc:1428] Setting key 524a9f09997275b508aeae0fc1f4a182
Sep 25 21:56:18 dist-test-slave-2g0f krb5kdc[5857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:18.438498 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:18.429849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:40585 (local address 127.2.70.254:46253)
0925 21:56:18.429906 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:18.429908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:18.430101 (+ 193us) server_negotiation.cc:407] Connection header received
0925 21:56:18.431060 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:18.431061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:18.431074 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:18.431146 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:18.431953 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.432451 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.433507 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.433726 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.433789 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:18.435877 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:18.435879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:18.435879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:18.435887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:18.437119 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:18.437457 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:18.437458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:18.437458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:18.437506 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:18.437813 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:18.437814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:18.437814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:18.437931 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:18.437988 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:18.438356 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 21:56:18.438431 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 21:56:18.438632 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:18.429826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46253 (local address 127.2.70.195:40585)
0925 21:56:18.429974 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:18.430016 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:18.430963 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:18.431232 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:18.431237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:18.431591 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:18.431893 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:18.431899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.432531 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.432532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:18.433408 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:18.433411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.433879 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.433880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:18.433995 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:18.434425 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:18.434432 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:18.435769 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:18.437191 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:18.437196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:18.437198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:18.437400 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:18.437584 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:18.437585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:18.437585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:18.437711 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:18.437987 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:18.437990 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:18.438176 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 21:56:18.438294 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 21:56:18.438437 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260925 21:56:18.439163 6361 heartbeater.cc:347] Connected to a master server at 127.2.70.254:46253
I20260925 21:56:18.439304 6361 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:18.439456 6361 heartbeater.cc:511] Master 127.2.70.254:46253 requested a full tablet report, sending...
I20260925 21:56:18.439790 5902 ts_manager.cc:194] Registered new tserver with Master: 52a075ece0c74721b9d38552c13fb171 (127.2.70.195:37649)
I20260925 21:56:18.440171 5902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:40585
I20260925 21:56:18.442535 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 21:56:18 dist-test-slave-2g0f krb5kdc[5857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 21:56:18 dist-test-slave-2g0f krb5kdc[5857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.193@KRBTEST.COM
I20260925 21:56:18.464656 6368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:18.457295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.2.70.193:40101)
0925 21:56:18.457457 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:18.457463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:18.457472 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:18.457497 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:18.457499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:18.457512 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:18.457577 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:18.458113 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.458834 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.459541 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.459902 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.459958 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:18.462006 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:18.462013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:18.462013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:18.462020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:18.463495 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:18.463824 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:18.463846 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:18.463847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:18.463906 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:18.464077 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:18.464078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:18.464078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:18.464240 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:18.464346 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:18.464480 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 21:56:18.464550 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
Sep 25 21:56:18 dist-test-slave-2g0f krb5kdc[5857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:18.472178 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:18.465915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.2.70.254:46253)
0925 21:56:18.465972 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:18.465974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:18.466010 (+ 36us) server_negotiation.cc:407] Connection header received
0925 21:56:18.466062 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:18.466063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:18.466073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:18.466133 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:18.466530 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.467003 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.467660 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:18.467936 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:18.467994 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:18.469887 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:18.469888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:18.469888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:18.469897 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:18.471156 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:18.471462 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:18.471463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:18.471463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:18.471523 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:18.471655 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:18.471655 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:18.471655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:18.471797 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:18.471904 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:18.471977 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 21:56:18.472041 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 21:56:18.472421 5902 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37314
I20260925 21:56:18.472738 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 5964
I20260925 21:56:18.477305 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 6098
I20260925 21:56:18.481025 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 6232
I20260925 21:56:18.484501 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 5870
2026-09-25T21:56:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1716 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:18 dist-test-slave-2g0f krb5kdc[6373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:18 dist-test-slave-2g0f krb5kdc[6373](info): set up 2 sockets
Sep 25 21:56:18 dist-test-slave-2g0f krb5kdc[6373](info): commencing operation
krb5kdc: starting...
W20260925 21:56:19.455551 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:19 dist-test-slave-2g0f krb5kdc[6373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:56:19.572543 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:35233
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:35233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:19.653071 6386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:19.653293 6386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:19.653326 6386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:19.654528 6386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:19.654572 6386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:19.654589 6386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:19.654605 6386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:19.654621 6386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:19.656172 6386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:35233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:35233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:19.656420 6386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:19.656627 6386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:19.661286 6392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:19.661340 6394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:19.661435 6391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:19.661504 6386 server_base.cc:1034] running on GCE node
I20260925 21:56:19.661783 6386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:19.662038 6386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:19.663180 6386 hybrid_clock.cc:648] HybridClock initialized: now 1790373379663164 us; error 29 us; skew 500 ppm
Sep 25 21:56:19 dist-test-slave-2g0f krb5kdc[6373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373379, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:19.665608 6386 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:56:19.665938 6386 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:19.665987 6386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:19.666086 6386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:19.667162 6386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "94b489f5ec3c41fb89d692dd7c215c96"
format_stamp: "Formatted at 2026-09-25 21:56:19 on dist-test-slave-2g0f"
server_key: "c87df5eb9bd9e57838164a45ccf12e42"
server_key_iv: "33bce431d17777de6630fc52216cb54b"
server_key_version: "encryptionkey@0"
I20260925 21:56:19.667528 6386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "94b489f5ec3c41fb89d692dd7c215c96"
format_stamp: "Formatted at 2026-09-25 21:56:19 on dist-test-slave-2g0f"
server_key: "c87df5eb9bd9e57838164a45ccf12e42"
server_key_iv: "33bce431d17777de6630fc52216cb54b"
server_key_version: "encryptionkey@0"
I20260925 21:56:19.668787 6386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:19.669502 6400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:19.669668 6386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:19.669751 6386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "94b489f5ec3c41fb89d692dd7c215c96"
format_stamp: "Formatted at 2026-09-25 21:56:19 on dist-test-slave-2g0f"
server_key: "c87df5eb9bd9e57838164a45ccf12e42"
server_key_iv: "33bce431d17777de6630fc52216cb54b"
server_key_version: "encryptionkey@0"
I20260925 21:56:19.669811 6386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:19.680738 6386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:19.682880 6386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:19.683017 6386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:19.686707 6386 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:35233
I20260925 21:56:19.686751 6452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:35233 every 8 connection(s)
I20260925 21:56:19.687019 6386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:19.687278 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 6386
I20260925 21:56:19.687373 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:19.687517 2331 external_mini_cluster.cc:1428] Setting key e257dfc1b1f3cf52123c606fe6db0468
I20260925 21:56:19.688158 6453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:19.691198 6453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96: Bootstrap starting.
I20260925 21:56:19.691944 6453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:19.692219 6453 log.cc:824] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:19.692968 6453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96: No bootstrap required, opened a new log
Sep 25 21:56:19 dist-test-slave-2g0f krb5kdc[6373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:19.695276 6453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b489f5ec3c41fb89d692dd7c215c96" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35233 } }
I20260925 21:56:19.695428 6453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:19.695465 6453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b489f5ec3c41fb89d692dd7c215c96, State: Initialized, Role: FOLLOWER
I20260925 21:56:19.695619 6453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b489f5ec3c41fb89d692dd7c215c96" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35233 } }
I20260925 21:56:19.695695 6453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:19.695730 6453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:19.695782 6453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:19.696444 6453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b489f5ec3c41fb89d692dd7c215c96" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35233 } }
I20260925 21:56:19.696563 6453 leader_election.cc:302] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94b489f5ec3c41fb89d692dd7c215c96; no voters:
I20260925 21:56:19.696738 6453 leader_election.cc:288] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:19.696782 6458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:19.696935 6458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [term 1 LEADER]: Becoming Leader. State: Replica: 94b489f5ec3c41fb89d692dd7c215c96, State: Running, Role: LEADER
I20260925 21:56:19.697023 6453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:19.697129 6458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b489f5ec3c41fb89d692dd7c215c96" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35233 } }
I20260925 21:56:19.697503 6458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94b489f5ec3c41fb89d692dd7c215c96. Latest consensus state: current_term: 1 leader_uuid: "94b489f5ec3c41fb89d692dd7c215c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b489f5ec3c41fb89d692dd7c215c96" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35233 } } }
I20260925 21:56:19.697588 6458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:19.697731 6460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94b489f5ec3c41fb89d692dd7c215c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b489f5ec3c41fb89d692dd7c215c96" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35233 } } }
I20260925 21:56:19.697788 6460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:19.698633 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:19.688030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.2.70.254:35233)
0925 21:56:19.688192 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:19.688203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:19.688237 (+ 34us) server_negotiation.cc:407] Connection header received
0925 21:56:19.688707 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:19.688715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:19.688773 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:19.688890 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:19.689783 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:19.690661 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:19.691705 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:19.692224 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:19.692323 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:19.695094 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:19.695098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:19.695098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:19.695109 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:19.696814 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:19.697205 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:19.697209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:19.697209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:19.697325 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:19.697518 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:19.697519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:19.697520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:19.697819 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:19.697957 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:19.698266 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 21:56:19.698428 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
W20260925 21:56:19.698899 6474 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:56:19.698956 6474 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:56:19.699164 6466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:19.699337 6466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:19.700870 6466 catalog_manager.cc:1347] Generated new cluster ID: 95eee2c1e9714018a3faa51fbbd889c2
I20260925 21:56:19.700917 6466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:19.705127 6466 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:19.705799 6466 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:19.710920 6466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96: Generated new TSK 0
I20260925 21:56:19.711136 6466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:56:19.747218 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35233
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:19.828989 6479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:19.829213 6479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:19.829247 6479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:19.830435 6479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:19.830475 6479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:19.830513 6479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:19.832103 6479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:19.832340 6479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:19.832593 6479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:19.837512 6484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:19.837519 6487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:19.837524 6485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:19.837694 6479 server_base.cc:1034] running on GCE node
I20260925 21:56:19.837960 6479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:19.838258 6479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:19.839412 6479 hybrid_clock.cc:648] HybridClock initialized: now 1790373379839390 us; error 36 us; skew 500 ppm
Sep 25 21:56:19 dist-test-slave-2g0f krb5kdc[6373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373379, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:19.842002 6479 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:56:19.842411 6479 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:19.842470 6479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:19.842602 6479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:19.843752 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "2c02285a0172484794fc8a8bedaf72c8"
format_stamp: "Formatted at 2026-09-25 21:56:19 on dist-test-slave-2g0f"
server_key: "12482914a43502d6e7b373f46104f0a0"
server_key_iv: "ed9a9c437332a6786d370a1a298ee858"
server_key_version: "encryptionkey@0"
I20260925 21:56:19.844111 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "2c02285a0172484794fc8a8bedaf72c8"
format_stamp: "Formatted at 2026-09-25 21:56:19 on dist-test-slave-2g0f"
server_key: "12482914a43502d6e7b373f46104f0a0"
server_key_iv: "ed9a9c437332a6786d370a1a298ee858"
server_key_version: "encryptionkey@0"
I20260925 21:56:19.845350 6479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:19.846014 6493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:19.846163 6479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:19.846290 6479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "2c02285a0172484794fc8a8bedaf72c8"
format_stamp: "Formatted at 2026-09-25 21:56:19 on dist-test-slave-2g0f"
server_key: "12482914a43502d6e7b373f46104f0a0"
server_key_iv: "ed9a9c437332a6786d370a1a298ee858"
server_key_version: "encryptionkey@0"
I20260925 21:56:19.846349 6479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:19.859436 6479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:19.861456 6479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:19.861568 6479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:19.861768 6479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:19.862030 6479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:19.862063 6479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:19.862084 6479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:19.862102 6479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:19.867353 6479 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:45497
I20260925 21:56:19.867422 6606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:45497 every 8 connection(s)
I20260925 21:56:19.867648 6479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:19.872319 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 6479
I20260925 21:56:19.872440 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:19.872603 2331 external_mini_cluster.cc:1428] Setting key 3862033e8e1f28fccd9959de4b2eda8a
Sep 25 21:56:19 dist-test-slave-2g0f krb5kdc[6373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373379, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:19.877441 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:19.868385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:52911 (local address 127.2.70.254:35233)
0925 21:56:19.868436 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:19.868438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:19.868620 (+ 182us) server_negotiation.cc:407] Connection header received
0925 21:56:19.869126 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:19.869127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:19.869140 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:19.869191 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:19.870164 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:19.870674 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:19.871696 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:19.871964 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:19.872023 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:19.874612 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:19.874614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:19.874614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:19.874623 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:19.876074 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:19.876446 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:19.876447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:19.876448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:19.876509 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:19.876714 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:19.876715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:19.876715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:19.876870 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:19.876937 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:19.877248 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 21:56:19.877345 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 21:56:19.877521 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:19.868337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35233 (local address 127.2.70.193:52911)
0925 21:56:19.868501 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:19.868541 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:19.869023 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:19.869287 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:19.869292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:19.869555 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:19.870052 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:19.870058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:19.870768 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:19.870769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:19.871604 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:19.871607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:19.872044 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:19.872045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:19.872197 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:19.872904 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:19.872915 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:19.874515 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:19.876155 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:19.876162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:19.876164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:19.876381 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:19.876568 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:19.876569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:19.876569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:19.876663 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:19.876939 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:19.876943 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:19.877073 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 21:56:19.877210 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 21:56:19.877342 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260925 21:56:19.878010 6607 heartbeater.cc:347] Connected to a master server at 127.2.70.254:35233
I20260925 21:56:19.878110 6607 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:19.878273 6607 heartbeater.cc:511] Master 127.2.70.254:35233 requested a full tablet report, sending...
I20260925 21:56:19.878898 6417 ts_manager.cc:194] Registered new tserver with Master: 2c02285a0172484794fc8a8bedaf72c8 (127.2.70.193:45497)
I20260925 21:56:19.879817 6417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:52911
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:56:19.899684 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35233
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:19.979591 6612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:19.979802 6612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:19.979848 6612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:19.981050 6612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:19.981096 6612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:19.981124 6612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:19.982662 6612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:19.982894 6612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:19.983117 6612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:19.988183 6620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:19.988153 6618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:19.988246 6617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:19.988165 6612 server_base.cc:1034] running on GCE node
I20260925 21:56:19.988519 6612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:19.988750 6612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:19.989899 6612 hybrid_clock.cc:648] HybridClock initialized: now 1790373379989862 us; error 53 us; skew 500 ppm
Sep 25 21:56:19 dist-test-slave-2g0f krb5kdc[6373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373379, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:19.996194 6612 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:56:19.996580 6612 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:19.996675 6612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:19.996809 6612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:19.997957 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "85b46d2afc784055b98e58354ea02d40"
format_stamp: "Formatted at 2026-09-25 21:56:19 on dist-test-slave-2g0f"
server_key: "0458abdb865f8915d66d5577608c53bb"
server_key_iv: "c0d76be49de7f410f36a346b1c3d3219"
server_key_version: "encryptionkey@0"
I20260925 21:56:19.998289 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "85b46d2afc784055b98e58354ea02d40"
format_stamp: "Formatted at 2026-09-25 21:56:19 on dist-test-slave-2g0f"
server_key: "0458abdb865f8915d66d5577608c53bb"
server_key_iv: "c0d76be49de7f410f36a346b1c3d3219"
server_key_version: "encryptionkey@0"
I20260925 21:56:19.999576 6612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 21:56:20.000414 6626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.000592 6612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:56:20.000666 6612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "85b46d2afc784055b98e58354ea02d40"
format_stamp: "Formatted at 2026-09-25 21:56:19 on dist-test-slave-2g0f"
server_key: "0458abdb865f8915d66d5577608c53bb"
server_key_iv: "c0d76be49de7f410f36a346b1c3d3219"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.000741 6612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:20.015565 6612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:20.017427 6612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:20.017551 6612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:20.017782 6612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:20.018087 6612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:20.018121 6612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.018142 6612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:20.018153 6612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.023588 6612 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:34489
I20260925 21:56:20.023633 6739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:34489 every 8 connection(s)
I20260925 21:56:20.023917 6612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:20.024389 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 6612
I20260925 21:56:20.024495 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:20.024694 2331 external_mini_cluster.cc:1428] Setting key 2e7281f1ac75a33ffc477f5d4aa67991
Sep 25 21:56:20 dist-test-slave-2g0f krb5kdc[6373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373379, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
I20260925 21:56:20.034503 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.024616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:41617 (local address 127.2.70.254:35233)
0925 21:56:20.024676 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.024679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.024962 (+ 283us) server_negotiation.cc:407] Connection header received
0925 21:56:20.025532 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.025533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.025543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.025575 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.026795 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.027319 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.028587 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.028866 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.028946 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.031417 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.031419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.031420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:20.031428 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.032988 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:20.033436 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:20.033437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:20.033438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:20.033493 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:20.033776 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:20.033778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:20.033778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:20.033933 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.034007 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.034322 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.034411 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 21:56:20.034694 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.024617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35233 (local address 127.2.70.194:41617)
0925 21:56:20.024815 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.024866 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.025432 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.025665 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.025669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.026030 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:20.026683 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.026691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.027418 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.027420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.028492 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.028495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.028945 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.028946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.029054 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.029647 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:20.029656 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:20.031311 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:20.033083 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:20.033089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:20.033092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:20.033364 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:20.033547 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:20.033547 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:20.033548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:20.033659 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:20.034028 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:20.034035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:20.034181 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.034277 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.034456 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260925 21:56:20.035181 6740 heartbeater.cc:347] Connected to a master server at 127.2.70.254:35233
I20260925 21:56:20.035274 6740 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:20.035425 6740 heartbeater.cc:511] Master 127.2.70.254:35233 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
I20260925 21:56:20.035859 6417 ts_manager.cc:194] Registered new tserver with Master: 85b46d2afc784055b98e58354ea02d40 (127.2.70.194:34489)
I20260925 21:56:20.036391 6417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:41617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:56:20.050946 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35233
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:20.130857 6745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:20.131081 6745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:20.131115 6745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:20.132339 6745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:20.132378 6745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:20.132413 6745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:20.133937 6745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:20.134187 6745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:20.134402 6745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:20.139475 6753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:20.139504 6750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:20.139602 6745 server_base.cc:1034] running on GCE node
W20260925 21:56:20.139500 6751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:20.139940 6745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:20.140230 6745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:20.141398 6745 hybrid_clock.cc:648] HybridClock initialized: now 1790373380141341 us; error 68 us; skew 500 ppm
Sep 25 21:56:20 dist-test-slave-2g0f krb5kdc[6373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373380, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:20.143891 6745 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:56:20.144284 6745 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:20.144337 6745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:20.144456 6745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:20.145535 6745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "5af79b04e95a421ba04ff24c2ced12ce"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "354d6a3198f7fdcdb721f4d2e5220bc3"
server_key_iv: "6b9260aefe90769de90e9416d2fb4127"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.145880 6745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "5af79b04e95a421ba04ff24c2ced12ce"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "354d6a3198f7fdcdb721f4d2e5220bc3"
server_key_iv: "6b9260aefe90769de90e9416d2fb4127"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.147142 6745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:20.147948 6759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.148105 6745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:20.148238 6745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "5af79b04e95a421ba04ff24c2ced12ce"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "354d6a3198f7fdcdb721f4d2e5220bc3"
server_key_iv: "6b9260aefe90769de90e9416d2fb4127"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.148303 6745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:20.156598 6745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:20.158327 6745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:20.158464 6745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:20.158672 6745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:20.158970 6745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:20.159003 6745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.159030 6745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:20.159047 6745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.164412 6745 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:40463
I20260925 21:56:20.164470 6872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:40463 every 8 connection(s)
I20260925 21:56:20.164708 6745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:20.165504 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 6745
I20260925 21:56:20.165634 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:20.165833 2331 external_mini_cluster.cc:1428] Setting key 1f67401bb2ddd7e79d0bdef8cf0821e9
Sep 25 21:56:20 dist-test-slave-2g0f krb5kdc[6373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373380, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:20.173727 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.165453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:48181 (local address 127.2.70.254:35233)
0925 21:56:20.165524 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.165526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.165648 (+ 122us) server_negotiation.cc:407] Connection header received
0925 21:56:20.166192 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.166193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.166205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.166244 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.167284 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.167766 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.168755 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.168948 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.169001 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.171156 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.171157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.171157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:20.171164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.172456 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:20.172870 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:20.172872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:20.172872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:20.172923 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:20.173134 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:20.173135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:20.173135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:20.173246 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.173300 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.173553 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.173633 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 21:56:20.173920 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.165377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35233 (local address 127.2.70.195:48181)
0925 21:56:20.165535 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.165575 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.166104 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.166327 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.166331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.166680 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:20.167207 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.167213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.167870 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.167871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.168682 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.168684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.169021 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.169022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.169124 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.169625 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:20.169634 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:20.171017 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:20.172534 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:20.172541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:20.172543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:20.172802 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:20.172977 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:20.172977 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:20.172978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:20.173078 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:20.173300 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:20.173306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:20.173439 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.173528 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.173697 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260925 21:56:20.174365 6873 heartbeater.cc:347] Connected to a master server at 127.2.70.254:35233
I20260925 21:56:20.174450 6873 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:20.174590 6873 heartbeater.cc:511] Master 127.2.70.254:35233 requested a full tablet report, sending...
I20260925 21:56:20.174952 6417 ts_manager.cc:194] Registered new tserver with Master: 5af79b04e95a421ba04ff24c2ced12ce (127.2.70.195:40463)
I20260925 21:56:20.175325 6417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:48181
I20260925 21:56:20.177338 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:56:20.183993 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.178567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.2.70.254:35233)
0925 21:56:20.178609 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.178611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.178700 (+ 89us) server_negotiation.cc:407] Connection header received
0925 21:56:20.178738 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.178739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.178748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.178794 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.179366 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.179868 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.180571 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.180762 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.180819 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.181890 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.181891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.181892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:20.181898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.183021 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:20.183352 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:20.183353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:20.183353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:20.183402 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:20.183529 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:20.183530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:20.183530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:20.183660 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.183734 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.183854 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.183926 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 21:56:20.185384 6417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:20.187036 6417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:20.193158 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.189220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.2.70.193:45497)
0925 21:56:20.189402 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.189406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.189416 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:20.189477 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.189479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.189494 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.189573 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:20.189846 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.190997 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.192264 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.192781 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.192854 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.192917 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.193038 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.193076 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260925 21:56:20.193183 6884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.189029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:45497 (local address 127.0.0.1:43318)
0925 21:56:20.189346 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.189357 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.189410 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.189585 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.189588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.189590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:20.189663 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.189664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.191095 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.191096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.192197 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.192199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.192856 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.192857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.192936 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.192949 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.193012 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.193066 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":37,"thread_start_us":54,"threads_started":1}
I20260925 21:56:20.193280 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.189113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.2.70.195:40463)
0925 21:56:20.189620 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.189624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.189633 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:20.189667 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.189669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.189684 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.189772 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:20.189948 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.190710 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.192135 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.192825 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.192880 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.192925 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.193162 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.193204 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":105,"threads_started":1}
I20260925 21:56:20.193401 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.189029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:40463 (local address 127.0.0.1:50166)
0925 21:56:20.189186 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.189226 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.189323 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.189840 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.189841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.189843 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:20.189877 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.189878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.190810 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.190811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.192059 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.192061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.192960 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.192961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.193036 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.193039 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.193090 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.193124 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":56,"threads_started":1}
I20260925 21:56:20.193547 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.189215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:34489 (local address 127.0.0.1:47758)
0925 21:56:20.189417 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.189425 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.189462 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.189720 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.189721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.189723 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:20.189767 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.189769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.191109 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.191109 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.191991 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.191994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.192962 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.192963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.193028 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.193032 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.193077 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.193117 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":24,"thread_start_us":85,"threads_started":1}
I20260925 21:56:20.193737 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.189431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.2.70.194:34489)
0925 21:56:20.189587 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.189591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.189600 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:20.189636 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.189637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.189651 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.189717 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:20.189818 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.191009 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.192072 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.192797 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.192867 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.192925 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.193428 (+ 503us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.193470 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260925 21:56:20.194123 6807 tablet_service.cc:1467] Processing CreateTablet for tablet b11675fa22954364ad596f515e27f052 (DEFAULT_TABLE table=test-table [id=23b8238c71784e90b8f87f1e4629bb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:20.194123 6541 tablet_service.cc:1467] Processing CreateTablet for tablet b11675fa22954364ad596f515e27f052 (DEFAULT_TABLE table=test-table [id=23b8238c71784e90b8f87f1e4629bb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:20.194483 6807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b11675fa22954364ad596f515e27f052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:20.194483 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b11675fa22954364ad596f515e27f052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:20.196496 6674 tablet_service.cc:1467] Processing CreateTablet for tablet b11675fa22954364ad596f515e27f052 (DEFAULT_TABLE table=test-table [id=23b8238c71784e90b8f87f1e4629bb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:20.196803 6674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b11675fa22954364ad596f515e27f052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:20.197881 6888 tablet_bootstrap.cc:492] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8: Bootstrap starting.
I20260925 21:56:20.198100 6887 tablet_bootstrap.cc:492] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce: Bootstrap starting.
I20260925 21:56:20.199093 6887 tablet_bootstrap.cc:654] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:20.199668 6887 log.cc:824] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:20.200273 6888 tablet_bootstrap.cc:654] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:20.200357 6889 tablet_bootstrap.cc:492] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40: Bootstrap starting.
I20260925 21:56:20.200625 6887 tablet_bootstrap.cc:492] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce: No bootstrap required, opened a new log
I20260925 21:56:20.200699 6887 ts_tablet_manager.cc:1397] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 21:56:20.200788 6888 log.cc:824] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:20.200929 6889 tablet_bootstrap.cc:654] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:20.201457 6889 log.cc:824] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:20.202207 6889 tablet_bootstrap.cc:492] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40: No bootstrap required, opened a new log
I20260925 21:56:20.202275 6889 ts_tablet_manager.cc:1397] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:20.202610 6887 raft_consensus.cc:348] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c02285a0172484794fc8a8bedaf72c8" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45497 } } peers { permanent_uuid: "85b46d2afc784055b98e58354ea02d40" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 34489 } } peers { permanent_uuid: "5af79b04e95a421ba04ff24c2ced12ce" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40463 } }
I20260925 21:56:20.202781 6887 raft_consensus.cc:374] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:20.202813 6887 raft_consensus.cc:729] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af79b04e95a421ba04ff24c2ced12ce, State: Initialized, Role: FOLLOWER
I20260925 21:56:20.202934 6887 consensus_queue.cc:260] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c02285a0172484794fc8a8bedaf72c8" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45497 } } peers { permanent_uuid: "85b46d2afc784055b98e58354ea02d40" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 34489 } } peers { permanent_uuid: "5af79b04e95a421ba04ff24c2ced12ce" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40463 } }
I20260925 21:56:20.203202 6887 ts_tablet_manager.cc:1428] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 21:56:20.203205 6873 heartbeater.cc:503] Master 127.2.70.254:35233 was elected leader, sending a full tablet report...
I20260925 21:56:20.203890 6888 tablet_bootstrap.cc:492] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8: No bootstrap required, opened a new log
I20260925 21:56:20.203959 6888 ts_tablet_manager.cc:1397] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 21:56:20.204664 6889 raft_consensus.cc:348] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c02285a0172484794fc8a8bedaf72c8" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45497 } } peers { permanent_uuid: "85b46d2afc784055b98e58354ea02d40" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 34489 } } peers { permanent_uuid: "5af79b04e95a421ba04ff24c2ced12ce" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40463 } }
I20260925 21:56:20.204840 6889 raft_consensus.cc:374] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:20.204883 6889 raft_consensus.cc:729] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b46d2afc784055b98e58354ea02d40, State: Initialized, Role: FOLLOWER
I20260925 21:56:20.204993 6889 consensus_queue.cc:260] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c02285a0172484794fc8a8bedaf72c8" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45497 } } peers { permanent_uuid: "85b46d2afc784055b98e58354ea02d40" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 34489 } } peers { permanent_uuid: "5af79b04e95a421ba04ff24c2ced12ce" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40463 } }
I20260925 21:56:20.205264 6889 ts_tablet_manager.cc:1428] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:56:20.205272 6740 heartbeater.cc:503] Master 127.2.70.254:35233 was elected leader, sending a full tablet report...
I20260925 21:56:20.205868 6888 raft_consensus.cc:348] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c02285a0172484794fc8a8bedaf72c8" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45497 } } peers { permanent_uuid: "85b46d2afc784055b98e58354ea02d40" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 34489 } } peers { permanent_uuid: "5af79b04e95a421ba04ff24c2ced12ce" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40463 } }
I20260925 21:56:20.206027 6888 raft_consensus.cc:374] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:20.206071 6888 raft_consensus.cc:729] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c02285a0172484794fc8a8bedaf72c8, State: Initialized, Role: FOLLOWER
I20260925 21:56:20.206185 6888 consensus_queue.cc:260] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c02285a0172484794fc8a8bedaf72c8" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45497 } } peers { permanent_uuid: "85b46d2afc784055b98e58354ea02d40" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 34489 } } peers { permanent_uuid: "5af79b04e95a421ba04ff24c2ced12ce" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40463 } }
I20260925 21:56:20.206441 6888 ts_tablet_manager.cc:1428] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:56:20.206477 6607 heartbeater.cc:503] Master 127.2.70.254:35233 was elected leader, sending a full tablet report...
W20260925 21:56:20.274406 6741 tablet.cc:2369] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:20.276474 6893 raft_consensus.cc:482] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:20.276615 6893 raft_consensus.cc:504] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c02285a0172484794fc8a8bedaf72c8" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45497 } } peers { permanent_uuid: "85b46d2afc784055b98e58354ea02d40" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 34489 } } peers { permanent_uuid: "5af79b04e95a421ba04ff24c2ced12ce" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40463 } }
I20260925 21:56:20.276999 6893 leader_election.cc:288] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c02285a0172484794fc8a8bedaf72c8 (127.2.70.193:45497), 85b46d2afc784055b98e58354ea02d40 (127.2.70.194:34489)
I20260925 21:56:20.279513 6895 raft_consensus.cc:482] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:20.279631 6895 raft_consensus.cc:504] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c02285a0172484794fc8a8bedaf72c8" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45497 } } peers { permanent_uuid: "85b46d2afc784055b98e58354ea02d40" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 34489 } } peers { permanent_uuid: "5af79b04e95a421ba04ff24c2ced12ce" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40463 } }
I20260925 21:56:20.280292 6895 leader_election.cc:288] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85b46d2afc784055b98e58354ea02d40 (127.2.70.194:34489), 5af79b04e95a421ba04ff24c2ced12ce (127.2.70.195:40463)
I20260925 21:56:20.280972 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.277266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:45497 (local address 127.2.70.195:43359)
0925 21:56:20.277539 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.277552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.277646 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.277794 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.277795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.277796 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:20.277857 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.277858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.278694 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.278695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.279903 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.279906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.280714 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.280716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.280787 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.280791 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.280850 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.280888 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260925 21:56:20.280974 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.277361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:43359 (local address 127.2.70.193:45497)
0925 21:56:20.277446 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.277448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.277673 (+ 225us) server_negotiation.cc:407] Connection header received
0925 21:56:20.277707 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.277709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.277722 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.277790 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:20.277896 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.278582 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.279991 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.280564 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.280640 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.280651 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.280863 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.280904 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 21:56:20.281266 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b11675fa22954364ad596f515e27f052" candidate_uuid: "5af79b04e95a421ba04ff24c2ced12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c02285a0172484794fc8a8bedaf72c8" is_pre_election: true
I20260925 21:56:20.281383 6561 raft_consensus.cc:2454] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5af79b04e95a421ba04ff24c2ced12ce in term 0.
I20260925 21:56:20.281509 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.277362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:34489 (local address 127.2.70.195:57779)
0925 21:56:20.278167 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.278179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.278293 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.278472 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.278472 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.278474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:20.278529 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.278531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.279454 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.279455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.280531 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.280533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.281292 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.281294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.281357 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.281361 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.281406 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.281447 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":102,"threads_started":1}
I20260925 21:56:20.281589 6762 leader_election.cc:302] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c02285a0172484794fc8a8bedaf72c8, 5af79b04e95a421ba04ff24c2ced12ce; no voters:
I20260925 21:56:20.281652 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.277591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:57779 (local address 127.2.70.194:34489)
0925 21:56:20.277650 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.277652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.278235 (+ 583us) server_negotiation.cc:407] Connection header received
0925 21:56:20.278372 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.278374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.278387 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.278463 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:20.278602 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.279364 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.280631 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.281194 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.281255 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.281263 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.281578 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.281610 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 21:56:20.281737 6893 raft_consensus.cc:2781] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:20.281790 6893 raft_consensus.cc:482] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:20.281811 6893 raft_consensus.cc:3037] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:20.282670 6893 raft_consensus.cc:504] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c02285a0172484794fc8a8bedaf72c8" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45497 } } peers { permanent_uuid: "85b46d2afc784055b98e58354ea02d40" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 34489 } } peers { permanent_uuid: "5af79b04e95a421ba04ff24c2ced12ce" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40463 } }
I20260925 21:56:20.282792 6893 leader_election.cc:288] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [CANDIDATE]: Term 1 election: Requested vote from peers 2c02285a0172484794fc8a8bedaf72c8 (127.2.70.193:45497), 85b46d2afc784055b98e58354ea02d40 (127.2.70.194:34489)
I20260925 21:56:20.283041 6694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b11675fa22954364ad596f515e27f052" candidate_uuid: "5af79b04e95a421ba04ff24c2ced12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b46d2afc784055b98e58354ea02d40" is_pre_election: true
I20260925 21:56:20.283185 6694 raft_consensus.cc:2454] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5af79b04e95a421ba04ff24c2ced12ce in term 0.
I20260925 21:56:20.283299 6693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b11675fa22954364ad596f515e27f052" candidate_uuid: "5af79b04e95a421ba04ff24c2ced12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b46d2afc784055b98e58354ea02d40"
I20260925 21:56:20.283349 6693 raft_consensus.cc:3037] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:20.283713 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.280151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:34489 (local address 127.2.70.193:45911)
0925 21:56:20.280673 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.280684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.280721 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.280873 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.280874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.280875 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:20.280927 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.280928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.281832 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.281833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.282889 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.282892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.283477 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.283478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.283559 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.283563 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.283604 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.283644 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":3}
I20260925 21:56:20.283723 6898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.280252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:45911 (local address 127.2.70.194:34489)
0925 21:56:20.280410 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.280412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.280767 (+ 355us) server_negotiation.cc:407] Connection header received
0925 21:56:20.280790 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.280791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.280799 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.280979 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:20.281012 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.281753 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.282978 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.283410 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.283470 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.283477 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.283617 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.283650 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260925 21:56:20.284207 6693 raft_consensus.cc:2454] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5af79b04e95a421ba04ff24c2ced12ce in term 1.
I20260925 21:56:20.284361 6762 leader_election.cc:302] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5af79b04e95a421ba04ff24c2ced12ce, 85b46d2afc784055b98e58354ea02d40; no voters:
I20260925 21:56:20.284438 6694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b11675fa22954364ad596f515e27f052" candidate_uuid: "2c02285a0172484794fc8a8bedaf72c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b46d2afc784055b98e58354ea02d40" is_pre_election: true
I20260925 21:56:20.284507 6893 raft_consensus.cc:2781] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:20.284526 6694 raft_consensus.cc:2379] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2c02285a0172484794fc8a8bedaf72c8 in current term 1: Already voted for candidate 5af79b04e95a421ba04ff24c2ced12ce in this term.
I20260925 21:56:20.284653 6893 raft_consensus.cc:686] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 1 LEADER]: Becoming Leader. State: Replica: 5af79b04e95a421ba04ff24c2ced12ce, State: Running, Role: LEADER
I20260925 21:56:20.284716 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.280666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:37431 (local address 127.2.70.195:40463)
0925 21:56:20.280704 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.280705 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.281929 (+ 1224us) server_negotiation.cc:407] Connection header received
0925 21:56:20.282005 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.282006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.282015 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.282040 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:20.282209 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.282635 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.283881 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.284370 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.284409 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.284417 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.284634 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.284670 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 21:56:20.284745 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.280151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:40463 (local address 127.2.70.193:37431)
0925 21:56:20.281880 (+ 1729us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.281890 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.281951 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.282088 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.282089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.282090 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:20.282151 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.282152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.282703 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.282704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.283757 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:20.283760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.284493 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.284494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:20.284562 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.284566 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.284612 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.284650 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1708,"mutex_wait_us":19,"thread_start_us":90,"threads_started":1}
I20260925 21:56:20.284739 6893 consensus_queue.cc:237] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c02285a0172484794fc8a8bedaf72c8" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45497 } } peers { permanent_uuid: "85b46d2afc784055b98e58354ea02d40" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 34489 } } peers { permanent_uuid: "5af79b04e95a421ba04ff24c2ced12ce" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40463 } }
I20260925 21:56:20.284952 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b11675fa22954364ad596f515e27f052" candidate_uuid: "5af79b04e95a421ba04ff24c2ced12ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c02285a0172484794fc8a8bedaf72c8"
I20260925 21:56:20.285022 6561 raft_consensus.cc:3037] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:20.285055 6827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b11675fa22954364ad596f515e27f052" candidate_uuid: "2c02285a0172484794fc8a8bedaf72c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af79b04e95a421ba04ff24c2ced12ce" is_pre_election: true
I20260925 21:56:20.285316 6494 leader_election.cc:302] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c02285a0172484794fc8a8bedaf72c8; no voters: 5af79b04e95a421ba04ff24c2ced12ce, 85b46d2afc784055b98e58354ea02d40
I20260925 21:56:20.285584 6416 catalog_manager.cc:5495] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 5af79b04e95a421ba04ff24c2ced12ce (127.2.70.195). New cstate: current_term: 1 leader_uuid: "5af79b04e95a421ba04ff24c2ced12ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c02285a0172484794fc8a8bedaf72c8" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85b46d2afc784055b98e58354ea02d40" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 34489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5af79b04e95a421ba04ff24c2ced12ce" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40463 } health_report { overall_health: HEALTHY } } }
I20260925 21:56:20.285817 6561 raft_consensus.cc:2454] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5af79b04e95a421ba04ff24c2ced12ce in term 1.
I20260925 21:56:20.285894 6895 raft_consensus.cc:2726] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 21:56:20.312402 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.309982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.2.70.195:40463)
0925 21:56:20.310050 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.310052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.310063 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:56:20.310106 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.310107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.310126 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.310193 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:20.310346 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.310840 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.311633 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:20.311851 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:20.311908 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:20.312076 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:20.312193 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:20.312268 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.312296 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.312317 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 21:56:20.314289 6694 raft_consensus.cc:1261] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40 [term 1 FOLLOWER]: Refusing update from remote peer 5af79b04e95a421ba04ff24c2ced12ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:20.314334 6561 raft_consensus.cc:1261] T b11675fa22954364ad596f515e27f052 P 2c02285a0172484794fc8a8bedaf72c8 [term 1 FOLLOWER]: Refusing update from remote peer 5af79b04e95a421ba04ff24c2ced12ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:20.314618 6893 consensus_queue.cc:1035] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b46d2afc784055b98e58354ea02d40" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:20.314759 6899 consensus_queue.cc:1035] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c02285a0172484794fc8a8bedaf72c8" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:20.317020 6902 mvcc.cc:205] Tried to move back new op lower bound from 7333369365764435968 to 7333369365647163392. Current Snapshot: MvccSnapshot[applied={T|T < 7333369365764435968}]
I20260925 21:56:20.317631 6904 mvcc.cc:205] Tried to move back new op lower bound from 7333369365764435968 to 7333369365647163392. Current Snapshot: MvccSnapshot[applied={T|T < 7333369365764435968}]
I20260925 21:56:20.319460 6417 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:20.319545 6417 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:20.320276 6417 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94b489f5ec3c41fb89d692dd7c215c96: Sending DeleteTablet for 3 replicas of tablet b11675fa22954364ad596f515e27f052
I20260925 21:56:20.320583 6807 tablet_service.cc:1514] Processing DeleteTablet for tablet b11675fa22954364ad596f515e27f052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:20 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:50166
I20260925 21:56:20.320608 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet b11675fa22954364ad596f515e27f052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:20 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:43318
I20260925 21:56:20.320808 6912 tablet_replica.cc:330] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce: stopping tablet replica
I20260925 21:56:20.320809 6674 tablet_service.cc:1514] Processing DeleteTablet for tablet b11675fa22954364ad596f515e27f052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:20 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:47758
I20260925 21:56:20.320933 6912 raft_consensus.cc:2229] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:20.320996 6914 tablet_replica.cc:330] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40: stopping tablet replica
I20260925 21:56:20.321059 6912 raft_consensus.cc:2258] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:20.321095 6914 raft_consensus.cc:2229] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:20.321203 6914 raft_consensus.cc:2258] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:20.321527 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 6479
I20260925 21:56:20.321550 6912 ts_tablet_manager.cc:1905] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:20.321607 6914 ts_tablet_manager.cc:1905] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:20.323401 6914 ts_tablet_manager.cc:1918] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:20.323405 6912 ts_tablet_manager.cc:1918] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:20.323475 6914 log.cc:1196] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/b11675fa22954364ad596f515e27f052
I20260925 21:56:20.323519 6912 log.cc:1196] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/b11675fa22954364ad596f515e27f052
I20260925 21:56:20.323803 6912 ts_tablet_manager.cc:1939] T b11675fa22954364ad596f515e27f052 P 5af79b04e95a421ba04ff24c2ced12ce: Deleting consensus metadata
I20260925 21:56:20.324056 6914 ts_tablet_manager.cc:1939] T b11675fa22954364ad596f515e27f052 P 85b46d2afc784055b98e58354ea02d40: Deleting consensus metadata
I20260925 21:56:20.324273 6401 catalog_manager.cc:4841] TS 5af79b04e95a421ba04ff24c2ced12ce (127.2.70.195:40463): tablet b11675fa22954364ad596f515e27f052 (table test-table [id=23b8238c71784e90b8f87f1e4629bb73]) successfully deleted
I20260925 21:56:20.324939 6403 catalog_manager.cc:4841] TS 85b46d2afc784055b98e58354ea02d40 (127.2.70.194:34489): tablet b11675fa22954364ad596f515e27f052 (table test-table [id=23b8238c71784e90b8f87f1e4629bb73]) successfully deleted
W20260925 21:56:20.327972 6403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:45497 (error 108)
I20260925 21:56:20.328423 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.328298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:45497 (local address 127.0.0.1:43320)
0925 21:56:20.328349 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.328388 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:45497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 21:56:20.328502 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 6612
W20260925 21:56:20.330750 6403 catalog_manager.cc:4584] TS 2c02285a0172484794fc8a8bedaf72c8 (127.2.70.193:45497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b11675fa22954364ad596f515e27f052: Network error: Client connection negotiation failed: client connection to 127.2.70.193:45497: connect: Connection refused (error 111)
I20260925 21:56:20.333518 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 6745
I20260925 21:56:20.337423 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 6386
2026-09-25T21:56:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1853 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T21:56:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:20Z Disabled control of system clock
I20260925 21:56:20.363121 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:42269
--webserver_interface=127.2.70.254
--webserver_port=0
--builtin_ntp_servers=127.2.70.212:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:42269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 21:56:20.444260 6919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:20.444479 6919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:20.444501 6919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:20.445674 6919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:20.445714 6919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:20.445726 6919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:20.445739 6919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:20.445748 6919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:20.447090 6919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:42269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:42269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:20.447297 6919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:20.447561 6919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:20.452351 6927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:20.452361 6919 server_base.cc:1034] running on GCE node
W20260925 21:56:20.452315 6925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:20.452342 6924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:20.452824 6919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:20.453155 6919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:20.454319 6919 hybrid_clock.cc:648] HybridClock initialized: now 1790373380454301 us; error 31 us; skew 500 ppm
I20260925 21:56:20.455552 6919 webserver.cc:460] Webserver started at http://127.2.70.254:44665/ using document root <none> and password file <none>
I20260925 21:56:20.455768 6919 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:20.455822 6919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:20.455979 6919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:20.457108 6919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "8ae560e1d1614cabb4c408719854abe1"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "81355a0100b5179d6488ac2e6b3b2a06"
server_key_iv: "6211775fc934b8eab039034b6b56492c"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.457424 6919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "8ae560e1d1614cabb4c408719854abe1"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "81355a0100b5179d6488ac2e6b3b2a06"
server_key_iv: "6211775fc934b8eab039034b6b56492c"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.458639 6919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 21:56:20.459290 6933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.459475 6919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 21:56:20.459560 6919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "8ae560e1d1614cabb4c408719854abe1"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "81355a0100b5179d6488ac2e6b3b2a06"
server_key_iv: "6211775fc934b8eab039034b6b56492c"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.459632 6919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:20.462940 6919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:20.463186 6919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:20.463301 6919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:20.467219 6919 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:42269
I20260925 21:56:20.467268 6985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:42269 every 8 connection(s)
I20260925 21:56:20.467559 6919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:20.468204 6986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:20.470904 6986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1: Bootstrap starting.
I20260925 21:56:20.471434 6986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:20.471665 6986 log.cc:824] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:20.472323 6986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1: No bootstrap required, opened a new log
I20260925 21:56:20.473847 6986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae560e1d1614cabb4c408719854abe1" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 42269 } }
I20260925 21:56:20.473986 6986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:20.474006 6986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae560e1d1614cabb4c408719854abe1, State: Initialized, Role: FOLLOWER
I20260925 21:56:20.474118 6986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae560e1d1614cabb4c408719854abe1" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 42269 } }
I20260925 21:56:20.474190 6986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:20.474220 6986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:20.474263 6986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:20.474751 6986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae560e1d1614cabb4c408719854abe1" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 42269 } }
I20260925 21:56:20.474833 6986 leader_election.cc:302] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ae560e1d1614cabb4c408719854abe1; no voters:
I20260925 21:56:20.474957 6986 leader_election.cc:288] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:20.475014 6989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:20.475179 6986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:20.475227 6989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [term 1 LEADER]: Becoming Leader. State: Replica: 8ae560e1d1614cabb4c408719854abe1, State: Running, Role: LEADER
I20260925 21:56:20.475317 6989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae560e1d1614cabb4c408719854abe1" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 42269 } }
I20260925 21:56:20.475708 6991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ae560e1d1614cabb4c408719854abe1. Latest consensus state: current_term: 1 leader_uuid: "8ae560e1d1614cabb4c408719854abe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae560e1d1614cabb4c408719854abe1" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 42269 } } }
I20260925 21:56:20.475782 6991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:20.475960 6990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ae560e1d1614cabb4c408719854abe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae560e1d1614cabb4c408719854abe1" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 42269 } } }
I20260925 21:56:20.476003 6998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:20.476027 6990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:20.476511 6998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:20.477358 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 6919
I20260925 21:56:20.477453 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:20.477654 2331 external_mini_cluster.cc:1428] Setting key ab1f702b2a9f3db74ea286044111002c
I20260925 21:56:20.478268 6998 catalog_manager.cc:1347] Generated new cluster ID: ee92ab2841ed4299af81621aac16206e
I20260925 21:56:20.478318 6998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:20.479791 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.478117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.2.70.254:42269)
0925 21:56:20.478285 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.478293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.478397 (+ 104us) server_negotiation.cc:407] Connection header received
0925 21:56:20.478515 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.478522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.478566 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.478715 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.479370 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.479374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.479374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:20.479378 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.479413 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:20.479427 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.479494 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.479548 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.479605 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260925 21:56:20.490438 6998 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:20.490622 6998 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:20.497182 6998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1: Generated new TSK 0
I20260925 21:56:20.497399 6998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:56:20.505856 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:42269
--builtin_ntp_servers=127.2.70.212:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 21:56:20.589510 7010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:20.589732 7010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:20.589771 7010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:20.590982 7010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:20.591027 7010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:20.591061 7010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:20.592520 7010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:20.592782 7010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:20.593072 7010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:20.597980 7015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:20.597973 7018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:20.597978 7016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:20.598268 7010 server_base.cc:1034] running on GCE node
I20260925 21:56:20.598501 7010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:20.598727 7010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:20.599882 7010 hybrid_clock.cc:648] HybridClock initialized: now 1790373380599870 us; error 33 us; skew 500 ppm
I20260925 21:56:20.601064 7010 webserver.cc:460] Webserver started at http://127.2.70.193:46767/ using document root <none> and password file <none>
I20260925 21:56:20.601289 7010 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:20.601338 7010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:20.601461 7010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:20.602550 7010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "2f5363e7889640849af99efa7f4985df"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "5b9e1527c49e64fafaf97d1b75664d07"
server_key_iv: "5a43b8254f643ecd6f44cd56a3625c59"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.602905 7010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "2f5363e7889640849af99efa7f4985df"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "5b9e1527c49e64fafaf97d1b75664d07"
server_key_iv: "5a43b8254f643ecd6f44cd56a3625c59"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.604218 7010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:20.604918 7024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.605082 7010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:20.605165 7010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "2f5363e7889640849af99efa7f4985df"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "5b9e1527c49e64fafaf97d1b75664d07"
server_key_iv: "5a43b8254f643ecd6f44cd56a3625c59"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.605224 7010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:20.608541 7010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:20.608811 7010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:20.608929 7010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:20.609112 7010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:20.609395 7010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:20.609427 7010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.609454 7010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:20.609480 7010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.615126 7010 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:41069
I20260925 21:56:20.615173 7137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:41069 every 8 connection(s)
I20260925 21:56:20.615489 7010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:20.618319 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.616226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:49427 (local address 127.2.70.254:42269)
0925 21:56:20.616303 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.616305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.616519 (+ 214us) server_negotiation.cc:407] Connection header received
0925 21:56:20.617003 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.617004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.617010 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.617047 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.618153 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.618154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.618154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:20.618157 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.618167 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:20.618170 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.618209 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.618255 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.618267 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 21:56:20.618530 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.616188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:42269 (local address 127.2.70.193:49427)
0925 21:56:20.616403 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.616440 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.616908 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.617111 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.617114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.617123 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:20.618021 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:20.618022 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:20.618052 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:20.618085 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:20.618208 (+ 123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:20.618211 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:20.618213 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.618270 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.618320 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
I20260925 21:56:20.619014 7138 heartbeater.cc:347] Connected to a master server at 127.2.70.254:42269
I20260925 21:56:20.619107 7138 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:20.619236 7138 heartbeater.cc:511] Master 127.2.70.254:42269 requested a full tablet report, sending...
I20260925 21:56:20.619764 6950 ts_manager.cc:194] Registered new tserver with Master: 2f5363e7889640849af99efa7f4985df (127.2.70.193:41069)
I20260925 21:56:20.620918 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 7010
I20260925 21:56:20.621001 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:20.621176 2331 external_mini_cluster.cc:1428] Setting key 71b43f0deeb44ed0d0d357315f4c672d
I20260925 21:56:20.621986 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:42269
--builtin_ntp_servers=127.2.70.212:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 21:56:20.702035 7141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:20.702246 7141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:20.702270 7141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:20.703471 7141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:20.703512 7141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:20.703538 7141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:20.704993 7141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:20.705233 7141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:20.705523 7141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:20.710467 7147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:20.710479 7141 server_base.cc:1034] running on GCE node
W20260925 21:56:20.710592 7146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:20.710501 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:20.710879 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:20.711150 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:20.712296 7141 hybrid_clock.cc:648] HybridClock initialized: now 1790373380712282 us; error 31 us; skew 500 ppm
I20260925 21:56:20.713506 7141 webserver.cc:460] Webserver started at http://127.2.70.194:36993/ using document root <none> and password file <none>
I20260925 21:56:20.713801 7141 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:20.713869 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:20.714087 7141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:20.715157 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "df5e39ee6b684cda9fb6eacc7690b727"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "204cd37d585c49ca64fe6c2e9c53e926"
server_key_iv: "73de7ceb6a00357287d5a81756354e27"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.715453 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "df5e39ee6b684cda9fb6eacc7690b727"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "204cd37d585c49ca64fe6c2e9c53e926"
server_key_iv: "73de7ceb6a00357287d5a81756354e27"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.716777 7141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:20.717483 7155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.717661 7141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:20.717767 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "df5e39ee6b684cda9fb6eacc7690b727"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "204cd37d585c49ca64fe6c2e9c53e926"
server_key_iv: "73de7ceb6a00357287d5a81756354e27"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.717828 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:20.721156 7141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:20.721410 7141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:20.721504 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:20.721663 7141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:20.721946 7141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:20.721979 7141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.721998 7141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:20.722009 7141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.727419 7141 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:42865
I20260925 21:56:20.727458 7268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:42865 every 8 connection(s)
I20260925 21:56:20.727772 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:20.730711 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.728569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:35661 (local address 127.2.70.254:42269)
0925 21:56:20.728662 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.728664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.728862 (+ 198us) server_negotiation.cc:407] Connection header received
0925 21:56:20.729375 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.729376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.729380 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.729432 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.730515 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.730517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.730517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:20.730521 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.730536 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:20.730541 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.730587 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.730638 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.730650 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 21:56:20.730890 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.728553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:42269 (local address 127.2.70.194:35661)
0925 21:56:20.728751 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.728784 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.729283 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.729495 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.729498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.729505 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:20.730372 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:20.730372 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:20.730401 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:20.730430 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:20.730585 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:20.730588 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:20.730590 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.730639 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.730693 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
I20260925 21:56:20.731328 7269 heartbeater.cc:347] Connected to a master server at 127.2.70.254:42269
I20260925 21:56:20.731406 7269 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:20.731527 7269 heartbeater.cc:511] Master 127.2.70.254:42269 requested a full tablet report, sending...
I20260925 21:56:20.731931 6950 ts_manager.cc:194] Registered new tserver with Master: df5e39ee6b684cda9fb6eacc7690b727 (127.2.70.194:42865)
I20260925 21:56:20.736747 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 7141
I20260925 21:56:20.736830 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:20.736963 2331 external_mini_cluster.cc:1428] Setting key 0a66f957727663e04ed44604b679c30c
I20260925 21:56:20.737789 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:42269
--builtin_ntp_servers=127.2.70.212:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 21:56:20.818342 7272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:20.818552 7272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:20.818571 7272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:20.819998 7272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:20.820050 7272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:20.820076 7272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:20.821544 7272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:20.821755 7272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:20.822052 7272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:20.826956 7280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:20.826948 7277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:20.827004 7272 server_base.cc:1034] running on GCE node
W20260925 21:56:20.826959 7278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:20.827416 7272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:20.827646 7272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:20.828789 7272 hybrid_clock.cc:648] HybridClock initialized: now 1790373380828775 us; error 24 us; skew 500 ppm
I20260925 21:56:20.830031 7272 webserver.cc:460] Webserver started at http://127.2.70.195:37517/ using document root <none> and password file <none>
I20260925 21:56:20.830322 7272 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:20.830371 7272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:20.830494 7272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:20.831569 7272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "0eebf0b1af544f0a89328cf4e730d5cd"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "186d676de12b2ae0d48c929511c82181"
server_key_iv: "b415c3e2b047335e6ba99d09e13b2527"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.831943 7272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "0eebf0b1af544f0a89328cf4e730d5cd"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "186d676de12b2ae0d48c929511c82181"
server_key_iv: "b415c3e2b047335e6ba99d09e13b2527"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.833222 7272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:20.833933 7286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.834149 7272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:20.834230 7272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "0eebf0b1af544f0a89328cf4e730d5cd"
format_stamp: "Formatted at 2026-09-25 21:56:20 on dist-test-slave-2g0f"
server_key: "186d676de12b2ae0d48c929511c82181"
server_key_iv: "b415c3e2b047335e6ba99d09e13b2527"
server_key_version: "encryptionkey@0"
I20260925 21:56:20.834296 7272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:20.837638 7272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:20.837881 7272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:20.837998 7272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:20.838194 7272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:20.838483 7272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:20.838515 7272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.838541 7272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:20.838557 7272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:20.844004 7272 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:40095
I20260925 21:56:20.844055 7399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:40095 every 8 connection(s)
I20260925 21:56:20.844376 7272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:20.847281 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.845166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:35097 (local address 127.2.70.254:42269)
0925 21:56:20.845230 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.845232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.845433 (+ 201us) server_negotiation.cc:407] Connection header received
0925 21:56:20.845881 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.845882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.845887 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.845934 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.847038 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.847039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.847040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:20.847044 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.847059 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:20.847064 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.847164 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.847216 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.847228 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 21:56:20.847452 7402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.845060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:42269 (local address 127.2.70.195:35097)
0925 21:56:20.845317 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.845351 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.845802 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.845990 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.845993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.846001 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:20.846887 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:20.846887 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:20.846916 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:20.846947 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:20.847164 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:20.847167 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:20.847169 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.847217 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.847266 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260925 21:56:20.847939 7400 heartbeater.cc:347] Connected to a master server at 127.2.70.254:42269
I20260925 21:56:20.848031 7400 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:20.848153 7400 heartbeater.cc:511] Master 127.2.70.254:42269 requested a full tablet report, sending...
I20260925 21:56:20.848503 6950 ts_manager.cc:194] Registered new tserver with Master: 0eebf0b1af544f0a89328cf4e730d5cd (127.2.70.195:40095)
I20260925 21:56:20.852421 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 7272
I20260925 21:56:20.852516 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:20.852685 2331 external_mini_cluster.cc:1428] Setting key 32474d47cb0100cafea6b8bf3be20bab
I20260925 21:56:20.853492 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:56:20.855731 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.854769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.2.70.254:42269)
0925 21:56:20.854839 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.854841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.854922 (+ 81us) server_negotiation.cc:407] Connection header received
0925 21:56:20.854972 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.854973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.854977 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.855026 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.855563 (+ 537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.855564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.855564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:20.855568 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.855583 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:20.855587 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.855625 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.855656 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.855667 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 21:56:20.857336 6950 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:20.858843 6950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:20.862468 7408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.861020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:41069 (local address 127.0.0.1:44786)
0925 21:56:20.861177 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.861216 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.861269 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.861515 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.861521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.861529 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:20.862037 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:20.862037 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:20.862067 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:20.862090 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:20.862311 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:20.862317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:20.862320 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.862382 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.862395 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":55,"threads_started":1}
I20260925 21:56:20.862491 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.861153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44786 (local address 127.2.70.193:41069)
0925 21:56:20.861372 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.861377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.861387 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:20.861413 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.861415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.861424 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.861499 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.862171 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.862179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.862179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:20.862184 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.862219 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:20.862242 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.862299 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.862390 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.862405 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260925 21:56:20.862869 7409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.861026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:40095 (local address 127.0.0.1:56854)
0925 21:56:20.861422 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.861432 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.861474 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.862032 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.862033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.862036 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:20.862502 (+ 466us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:20.862502 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:20.862509 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:20.862520 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:20.862750 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:20.862751 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:20.862751 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.862791 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.862802 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":109,"threads_started":1}
I20260925 21:56:20.863057 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.861257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.2.70.195:40095)
0925 21:56:20.861488 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.861492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.861500 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:20.861521 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.861523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.861530 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.861738 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.862617 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.862622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.862623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:20.862627 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.862657 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:20.862680 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.862727 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.862957 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.862971 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20260925 21:56:20.863088 7412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.861256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.2.70.194:42865)
0925 21:56:20.861410 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.861414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.861604 (+ 190us) server_negotiation.cc:407] Connection header received
0925 21:56:20.861692 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.861695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.861706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.861755 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.862777 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.862783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.862784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:20.862788 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.862821 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:20.862847 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.862879 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.862986 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.863002 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260925 21:56:20.863255 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.861026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:42865 (local address 127.0.0.1:55828)
0925 21:56:20.861552 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.861560 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.861625 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.862050 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.862050 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.862053 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:20.862687 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:20.862688 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:20.862694 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:20.862705 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:20.862918 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:20.862919 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:20.862920 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.863183 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.863197 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260925 21:56:20.863201 7072 tablet_service.cc:1467] Processing CreateTablet for tablet 3e2ffa356eea44b49c04c1708985cfda (DEFAULT_TABLE table=test-table [id=385d84f929d14eb0987a1f1c5f64b199]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:20.863646 7072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e2ffa356eea44b49c04c1708985cfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:20.863776 7334 tablet_service.cc:1467] Processing CreateTablet for tablet 3e2ffa356eea44b49c04c1708985cfda (DEFAULT_TABLE table=test-table [id=385d84f929d14eb0987a1f1c5f64b199]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:20.863958 7203 tablet_service.cc:1467] Processing CreateTablet for tablet 3e2ffa356eea44b49c04c1708985cfda (DEFAULT_TABLE table=test-table [id=385d84f929d14eb0987a1f1c5f64b199]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:20.864137 7334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e2ffa356eea44b49c04c1708985cfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:20.864261 7203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e2ffa356eea44b49c04c1708985cfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:20.866647 7414 tablet_bootstrap.cc:492] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd: Bootstrap starting.
I20260925 21:56:20.866909 7415 tablet_bootstrap.cc:492] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df: Bootstrap starting.
I20260925 21:56:20.867153 7416 tablet_bootstrap.cc:492] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727: Bootstrap starting.
I20260925 21:56:20.867300 7414 tablet_bootstrap.cc:654] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:20.867553 7415 tablet_bootstrap.cc:654] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:20.867614 7414 log.cc:824] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:20.867879 7415 log.cc:824] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:20.867923 7416 tablet_bootstrap.cc:654] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:20.868196 7416 log.cc:824] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:20.868424 7414 tablet_bootstrap.cc:492] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd: No bootstrap required, opened a new log
I20260925 21:56:20.868487 7414 ts_tablet_manager.cc:1397] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:20.868767 7416 tablet_bootstrap.cc:492] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727: No bootstrap required, opened a new log
I20260925 21:56:20.868816 7416 ts_tablet_manager.cc:1397] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 21:56:20.870357 7414 raft_consensus.cc:348] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f5363e7889640849af99efa7f4985df" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41069 } } peers { permanent_uuid: "0eebf0b1af544f0a89328cf4e730d5cd" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40095 } } peers { permanent_uuid: "df5e39ee6b684cda9fb6eacc7690b727" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42865 } }
I20260925 21:56:20.870491 7414 raft_consensus.cc:374] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:20.870519 7414 raft_consensus.cc:729] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eebf0b1af544f0a89328cf4e730d5cd, State: Initialized, Role: FOLLOWER
I20260925 21:56:20.870637 7414 consensus_queue.cc:260] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f5363e7889640849af99efa7f4985df" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41069 } } peers { permanent_uuid: "0eebf0b1af544f0a89328cf4e730d5cd" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40095 } } peers { permanent_uuid: "df5e39ee6b684cda9fb6eacc7690b727" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42865 } }
I20260925 21:56:20.870710 7416 raft_consensus.cc:348] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f5363e7889640849af99efa7f4985df" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41069 } } peers { permanent_uuid: "0eebf0b1af544f0a89328cf4e730d5cd" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40095 } } peers { permanent_uuid: "df5e39ee6b684cda9fb6eacc7690b727" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42865 } }
I20260925 21:56:20.870872 7416 raft_consensus.cc:374] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:20.870911 7414 ts_tablet_manager.cc:1428] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 21:56:20.870921 7416 raft_consensus.cc:729] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5e39ee6b684cda9fb6eacc7690b727, State: Initialized, Role: FOLLOWER
I20260925 21:56:20.870936 7400 heartbeater.cc:503] Master 127.2.70.254:42269 was elected leader, sending a full tablet report...
I20260925 21:56:20.871032 7416 consensus_queue.cc:260] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f5363e7889640849af99efa7f4985df" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41069 } } peers { permanent_uuid: "0eebf0b1af544f0a89328cf4e730d5cd" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40095 } } peers { permanent_uuid: "df5e39ee6b684cda9fb6eacc7690b727" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42865 } }
I20260925 21:56:20.871289 7416 ts_tablet_manager.cc:1428] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 21:56:20.871397 7269 heartbeater.cc:503] Master 127.2.70.254:42269 was elected leader, sending a full tablet report...
I20260925 21:56:20.871538 7415 tablet_bootstrap.cc:492] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df: No bootstrap required, opened a new log
I20260925 21:56:20.871620 7415 ts_tablet_manager.cc:1397] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 21:56:20.873693 7415 raft_consensus.cc:348] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f5363e7889640849af99efa7f4985df" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41069 } } peers { permanent_uuid: "0eebf0b1af544f0a89328cf4e730d5cd" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40095 } } peers { permanent_uuid: "df5e39ee6b684cda9fb6eacc7690b727" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42865 } }
I20260925 21:56:20.873827 7415 raft_consensus.cc:374] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:20.873848 7415 raft_consensus.cc:729] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5363e7889640849af99efa7f4985df, State: Initialized, Role: FOLLOWER
I20260925 21:56:20.873942 7415 consensus_queue.cc:260] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f5363e7889640849af99efa7f4985df" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41069 } } peers { permanent_uuid: "0eebf0b1af544f0a89328cf4e730d5cd" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40095 } } peers { permanent_uuid: "df5e39ee6b684cda9fb6eacc7690b727" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42865 } }
I20260925 21:56:20.874179 7415 ts_tablet_manager.cc:1428] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 21:56:20.874290 7138 heartbeater.cc:503] Master 127.2.70.254:42269 was elected leader, sending a full tablet report...
I20260925 21:56:20.915496 7422 raft_consensus.cc:482] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:20.915639 7422 raft_consensus.cc:504] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f5363e7889640849af99efa7f4985df" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41069 } } peers { permanent_uuid: "0eebf0b1af544f0a89328cf4e730d5cd" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40095 } } peers { permanent_uuid: "df5e39ee6b684cda9fb6eacc7690b727" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42865 } }
I20260925 21:56:20.916030 7422 leader_election.cc:288] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eebf0b1af544f0a89328cf4e730d5cd (127.2.70.195:40095), df5e39ee6b684cda9fb6eacc7690b727 (127.2.70.194:42865)
I20260925 21:56:20.917500 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.916209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:40095 (local address 127.2.70.193:46873)
0925 21:56:20.916251 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.916263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.916308 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.916608 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.916609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.916613 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:20.917156 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:20.917157 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:20.917166 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:20.917181 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:20.917351 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:20.917352 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:20.917352 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.917408 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.917422 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 21:56:20.917529 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.916370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:46873 (local address 127.2.70.195:40095)
0925 21:56:20.916412 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.916414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.916423 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:20.916450 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.916452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.916457 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.916516 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.917264 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.917266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.917266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:20.917271 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.917287 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:20.917292 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.917347 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.917444 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.917460 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 21:56:20.917605 7412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.916372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:52819 (local address 127.2.70.194:42865)
0925 21:56:20.916420 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.916422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.916572 (+ 150us) server_negotiation.cc:407] Connection header received
0925 21:56:20.916608 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.916609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.916615 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.916694 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.917333 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.917335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.917335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:20.917340 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.917359 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:20.917363 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.917423 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.917519 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.917540 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 21:56:20.917611 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.916293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:42865 (local address 127.2.70.193:52819)
0925 21:56:20.916457 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.916466 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:20.916515 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:20.916696 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:20.916697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:20.916699 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:20.917240 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:20.917241 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:20.917249 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:20.917262 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:20.917441 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:20.917442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:20.917443 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 21:56:20.917515 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 21:56:20.917528 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
I20260925 21:56:20.917797 7354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e2ffa356eea44b49c04c1708985cfda" candidate_uuid: "2f5363e7889640849af99efa7f4985df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eebf0b1af544f0a89328cf4e730d5cd" is_pre_election: true
I20260925 21:56:20.917939 7354 raft_consensus.cc:2454] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f5363e7889640849af99efa7f4985df in term 0.
I20260925 21:56:20.918068 7223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e2ffa356eea44b49c04c1708985cfda" candidate_uuid: "2f5363e7889640849af99efa7f4985df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5e39ee6b684cda9fb6eacc7690b727" is_pre_election: true
I20260925 21:56:20.918223 7223 raft_consensus.cc:2454] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f5363e7889640849af99efa7f4985df in term 0.
I20260925 21:56:20.918210 7028 leader_election.cc:302] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eebf0b1af544f0a89328cf4e730d5cd, 2f5363e7889640849af99efa7f4985df; no voters:
I20260925 21:56:20.918337 7422 raft_consensus.cc:2781] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:20.918393 7422 raft_consensus.cc:482] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:20.918417 7422 raft_consensus.cc:3037] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:20.919135 7422 raft_consensus.cc:504] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f5363e7889640849af99efa7f4985df" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41069 } } peers { permanent_uuid: "0eebf0b1af544f0a89328cf4e730d5cd" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40095 } } peers { permanent_uuid: "df5e39ee6b684cda9fb6eacc7690b727" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42865 } }
I20260925 21:56:20.919263 7422 leader_election.cc:288] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [CANDIDATE]: Term 1 election: Requested vote from peers 0eebf0b1af544f0a89328cf4e730d5cd (127.2.70.195:40095), df5e39ee6b684cda9fb6eacc7690b727 (127.2.70.194:42865)
I20260925 21:56:20.919409 7354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e2ffa356eea44b49c04c1708985cfda" candidate_uuid: "2f5363e7889640849af99efa7f4985df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eebf0b1af544f0a89328cf4e730d5cd"
I20260925 21:56:20.919446 7223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e2ffa356eea44b49c04c1708985cfda" candidate_uuid: "2f5363e7889640849af99efa7f4985df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5e39ee6b684cda9fb6eacc7690b727"
I20260925 21:56:20.919513 7223 raft_consensus.cc:3037] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:20.919507 7354 raft_consensus.cc:3037] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:20.920447 7354 raft_consensus.cc:2454] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f5363e7889640849af99efa7f4985df in term 1.
I20260925 21:56:20.920447 7223 raft_consensus.cc:2454] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f5363e7889640849af99efa7f4985df in term 1.
I20260925 21:56:20.920594 7028 leader_election.cc:302] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eebf0b1af544f0a89328cf4e730d5cd, 2f5363e7889640849af99efa7f4985df; no voters:
I20260925 21:56:20.920673 7422 raft_consensus.cc:2781] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:20.920847 7422 raft_consensus.cc:686] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [term 1 LEADER]: Becoming Leader. State: Replica: 2f5363e7889640849af99efa7f4985df, State: Running, Role: LEADER
I20260925 21:56:20.920940 7422 consensus_queue.cc:237] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f5363e7889640849af99efa7f4985df" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41069 } } peers { permanent_uuid: "0eebf0b1af544f0a89328cf4e730d5cd" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40095 } } peers { permanent_uuid: "df5e39ee6b684cda9fb6eacc7690b727" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42865 } }
I20260925 21:56:20.921800 6950 catalog_manager.cc:5495] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f5363e7889640849af99efa7f4985df (127.2.70.193). New cstate: current_term: 1 leader_uuid: "2f5363e7889640849af99efa7f4985df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f5363e7889640849af99efa7f4985df" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0eebf0b1af544f0a89328cf4e730d5cd" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df5e39ee6b684cda9fb6eacc7690b727" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42865 } health_report { overall_health: UNKNOWN } } }
I20260925 21:56:20.925424 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.924460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.2.70.193:41069)
0925 21:56:20.924521 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:20.924523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:20.924532 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:20.924566 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:20.924567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:20.924582 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:20.924635 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:20.925256 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:20.925257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:20.925257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:20.925261 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:20.925276 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:20.925280 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:20.925325 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:20.925359 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 21:56:20.925369 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 21:56:20.927111 7354 raft_consensus.cc:1261] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd [term 1 FOLLOWER]: Refusing update from remote peer 2f5363e7889640849af99efa7f4985df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:20.927393 7223 raft_consensus.cc:1261] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727 [term 1 FOLLOWER]: Refusing update from remote peer 2f5363e7889640849af99efa7f4985df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:20.927469 7424 consensus_queue.cc:1035] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eebf0b1af544f0a89328cf4e730d5cd" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:20.927634 7422 consensus_queue.cc:1035] T 3e2ffa356eea44b49c04c1708985cfda P 2f5363e7889640849af99efa7f4985df [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5e39ee6b684cda9fb6eacc7690b727" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:20.930742 7429 mvcc.cc:205] Tried to move back new op lower bound from 7333369368274972672 to 7333369368252854272. Current Snapshot: MvccSnapshot[applied={T|T < 7333369368274972672}]
I20260925 21:56:20.932610 6950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:20.932695 6950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:20.933387 6950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ae560e1d1614cabb4c408719854abe1: Sending DeleteTablet for 3 replicas of tablet 3e2ffa356eea44b49c04c1708985cfda
I20260925 21:56:20.933678 7334 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e2ffa356eea44b49c04c1708985cfda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:20 UTC) from {username='slave'} at 127.0.0.1:56854
I20260925 21:56:20.933692 7203 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e2ffa356eea44b49c04c1708985cfda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:20 UTC) from {username='slave'} at 127.0.0.1:55828
I20260925 21:56:20.933890 7072 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e2ffa356eea44b49c04c1708985cfda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:20 UTC) from {username='slave'} at 127.0.0.1:44786
I20260925 21:56:20.934010 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 7010
W20260925 21:56:20.938696 6935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:41069 (error 108)
I20260925 21:56:20.938776 7437 tablet_replica.cc:330] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727: stopping tablet replica
I20260925 21:56:20.938896 7437 raft_consensus.cc:2229] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:20.939018 7437 raft_consensus.cc:2258] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:20.939270 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.939102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:41069 (local address 127.0.0.1:44808)
0925 21:56:20.939197 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.939236 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 21:56:20.939342 7436 tablet_replica.cc:330] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd: stopping tablet replica
I20260925 21:56:20.939347 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 7141
W20260925 21:56:20.939370 6935 catalog_manager.cc:4584] TS 2f5363e7889640849af99efa7f4985df (127.2.70.193:41069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e2ffa356eea44b49c04c1708985cfda: Network error: Client connection negotiation failed: client connection to 127.2.70.193:41069: connect: Connection refused (error 111)
I20260925 21:56:20.939445 7437 ts_tablet_manager.cc:1905] T 3e2ffa356eea44b49c04c1708985cfda P df5e39ee6b684cda9fb6eacc7690b727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:20.943077 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.942962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:42865 (local address 127.0.0.1:55840)
0925 21:56:20.943007 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.943035 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.194:42865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 21:56:20.943264 7436 raft_consensus.cc:2229] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:20.943427 7436 raft_consensus.cc:2258] T 3e2ffa356eea44b49c04c1708985cfda P 0eebf0b1af544f0a89328cf4e730d5cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:20.943549 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 7272
I20260925 21:56:20.947315 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 6919
I20260925 21:56:20.947486 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:20.947348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:40095 (local address 127.0.0.1:56864)
0925 21:56:20.947418 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:20.947452 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.195:40095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
2026-09-25T21:56:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (609 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 21:56:20.966022 2331 mini_oidc.cc:152] Starting JWKS server
I20260925 21:56:20.967046 2331 webserver.cc:460] Webserver started at https://127.0.0.1:38211/ using document root <none> and password file <none>
I20260925 21:56:20.967126 2331 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 21:56:20.967276 2331 webserver.cc:460] Webserver started at http://127.0.0.1:42485/ using document root <none> and password file <none>
2026-09-25T21:56:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:20Z Disabled control of system clock
I20260925 21:56:20.974826 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:35457
--webserver_interface=127.2.70.254
--webserver_port=0
--builtin_ntp_servers=127.2.70.212:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:35457
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38211/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/testchainca.cert with env {}
W20260925 21:56:21.056716 7445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:21.056941 7445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:21.056965 7445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:21.058133 7445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:21.058172 7445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:21.058187 7445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:21.058198 7445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:21.058210 7445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 21:56:21.058229 7445 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 21:56:21.058248 7445 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38211/jwks.json
I20260925 21:56:21.059891 7445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:35457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:35457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38211/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:21.060112 7445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:21.060286 7445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:56:21.065410 7445 server_base.cc:1034] running on GCE node
W20260925 21:56:21.065380 7451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:21.065373 7450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:21.065380 7453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:21.065946 7445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:21.066273 7445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:21.067417 7445 hybrid_clock.cc:648] HybridClock initialized: now 1790373381067402 us; error 32 us; skew 500 ppm
I20260925 21:56:21.068647 7445 webserver.cc:460] Webserver started at http://127.2.70.254:35079/ using document root <none> and password file <none>
I20260925 21:56:21.068894 7445 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:21.068945 7445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:21.069072 7445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:21.070111 7445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "7b5f69996b924dd8ba0722480704761f"
format_stamp: "Formatted at 2026-09-25 21:56:21 on dist-test-slave-2g0f"
server_key: "a7fdaf3596fbba6b20a0bb3283ef7a60"
server_key_iv: "bcbc6e63da5a08b0e79b87401b7cdac1"
server_key_version: "encryptionkey@0"
I20260925 21:56:21.070436 7445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "7b5f69996b924dd8ba0722480704761f"
format_stamp: "Formatted at 2026-09-25 21:56:21 on dist-test-slave-2g0f"
server_key: "a7fdaf3596fbba6b20a0bb3283ef7a60"
server_key_iv: "bcbc6e63da5a08b0e79b87401b7cdac1"
server_key_version: "encryptionkey@0"
I20260925 21:56:21.071673 7445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:21.072436 7459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:21.072582 7445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:21.072662 7445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "7b5f69996b924dd8ba0722480704761f"
format_stamp: "Formatted at 2026-09-25 21:56:21 on dist-test-slave-2g0f"
server_key: "a7fdaf3596fbba6b20a0bb3283ef7a60"
server_key_iv: "bcbc6e63da5a08b0e79b87401b7cdac1"
server_key_version: "encryptionkey@0"
I20260925 21:56:21.072728 7445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:21.090597 7445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:21.090937 7445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:21.091065 7445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:21.094954 7445 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:35457
I20260925 21:56:21.094992 7511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:35457 every 8 connection(s)
I20260925 21:56:21.095315 7445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:21.096300 7512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:21.098574 7512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f: Bootstrap starting.
I20260925 21:56:21.099174 7512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:21.099325 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 7445
I20260925 21:56:21.099431 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:21.099491 7512 log.cc:824] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:21.099623 2331 external_mini_cluster.cc:1428] Setting key 8dd7851fbcd190410a8a9118a9c5504a
I20260925 21:56:21.100199 7512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f: No bootstrap required, opened a new log
I20260925 21:56:21.102562 7512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5f69996b924dd8ba0722480704761f" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35457 } }
I20260925 21:56:21.102697 7512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:21.102717 7512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5f69996b924dd8ba0722480704761f, State: Initialized, Role: FOLLOWER
I20260925 21:56:21.102824 7512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5f69996b924dd8ba0722480704761f" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35457 } }
I20260925 21:56:21.102902 7512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:21.102931 7512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:21.102972 7512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:21.103645 7512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5f69996b924dd8ba0722480704761f" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35457 } }
I20260925 21:56:21.103754 7512 leader_election.cc:302] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b5f69996b924dd8ba0722480704761f; no voters:
I20260925 21:56:21.103921 7512 leader_election.cc:288] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:21.103958 7517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:21.104076 7517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [term 1 LEADER]: Becoming Leader. State: Replica: 7b5f69996b924dd8ba0722480704761f, State: Running, Role: LEADER
I20260925 21:56:21.104162 7517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5f69996b924dd8ba0722480704761f" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35457 } }
I20260925 21:56:21.104231 7512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:21.104461 7518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b5f69996b924dd8ba0722480704761f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5f69996b924dd8ba0722480704761f" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35457 } } }
I20260925 21:56:21.104537 7518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:21.104749 7521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:21.104789 7519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b5f69996b924dd8ba0722480704761f. Latest consensus state: current_term: 1 leader_uuid: "7b5f69996b924dd8ba0722480704761f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5f69996b924dd8ba0722480704761f" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35457 } } }
I20260925 21:56:21.104894 7519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:21.105258 7521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:21.106097 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:21.100168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.2.70.254:35457)
0925 21:56:21.100370 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:21.100379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:21.100430 (+ 51us) server_negotiation.cc:407] Connection header received
0925 21:56:21.101413 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:21.101421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:21.101484 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:21.101596 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:21.101821 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.102688 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.103749 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.104421 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.104546 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:21.105226 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:21.105230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:21.105230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:21.105463 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:21.105503 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:21.105517 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:21.105767 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:21.105799 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 21:56:21.105881 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260925 21:56:21.107178 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 21:56:21.107544 7521 catalog_manager.cc:1347] Generated new cluster ID: d9e58942c2224e83a9d0d57ea735059c
I20260925 21:56:21.107583 7521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:21.120615 7521 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:21.121258 7521 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:21.130827 7521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b5f69996b924dd8ba0722480704761f: Generated new TSK 0
I20260925 21:56:21.131036 7521 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 21:56:21.163968 7542 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 21:56:21.164080 7542 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 21:56:21.164287 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:21.141670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.2.70.254:35457)
0925 21:56:21.141723 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:21.141726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:21.141806 (+ 80us) server_negotiation.cc:407] Connection header received
0925 21:56:21.141914 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:21.141915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:21.141927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:21.141986 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0925 21:56:21.142132 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.142663 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.143432 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.143641 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.143700 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:21.143952 (+ 252us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 21:56:21.163997 (+ 20045us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 21:56:21.164188 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40258: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 21:56:21.164371 7516 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40258: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 21:56:21.164858 7541 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:21.141567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35457 (local address 127.0.0.1:40258)
0925 21:56:21.141740 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:21.141750 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:21.141819 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:21.142002 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:21.142003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:21.142005 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 21:56:21.142075 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:21.142078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.142740 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.142742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:21.143352 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:21.143354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.143709 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.143710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:21.143777 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:21.143800 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 21:56:21.164284 (+ 20484us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 21:56:21.164411 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.70.254:35457: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.70.254:35457: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 21:56:21.165343 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 7445
2026-09-25T21:56:21Z chronyd exiting
I20260925 21:56:21.201232 2331 test_util.cc:175] -----------------------------------------------
I20260925 21:56:21.201318 2331 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790373357410105-2331-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (236 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 21:56:21.202113 2331 mini_oidc.cc:152] Starting JWKS server
I20260925 21:56:21.202790 2331 webserver.cc:460] Webserver started at https://127.0.0.1:37229/ using document root <none> and password file <none>
I20260925 21:56:21.202836 2331 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 21:56:21.202965 2331 webserver.cc:460] Webserver started at http://127.0.0.1:33347/ using document root <none> and password file <none>
2026-09-25T21:56:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:21Z Disabled control of system clock
I20260925 21:56:21.210274 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:33607
--webserver_interface=127.2.70.254
--webserver_port=0
--builtin_ntp_servers=127.2.70.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:33607
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37229/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/testchainca.cert with env {}
W20260925 21:56:21.291106 7548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:21.291344 7548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:21.291379 7548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:21.292711 7548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:21.292762 7548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:21.292779 7548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:21.292796 7548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:21.292811 7548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 21:56:21.292836 7548 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 21:56:21.292858 7548 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37229/jwks.json
I20260925 21:56:21.294510 7548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:33607
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:33607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37229/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:21.294781 7548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:21.294989 7548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:21.299860 7553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:21.299995 7548 server_base.cc:1034] running on GCE node
W20260925 21:56:21.299825 7556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:21.299820 7554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:21.300329 7548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:21.300599 7548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:21.301744 7548 hybrid_clock.cc:648] HybridClock initialized: now 1790373381301727 us; error 28 us; skew 500 ppm
I20260925 21:56:21.302975 7548 webserver.cc:460] Webserver started at http://127.2.70.254:33653/ using document root <none> and password file <none>
I20260925 21:56:21.303228 7548 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:21.303274 7548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:21.303365 7548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:21.304457 7548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "557be3a1439548f6ad2e04c9446645e0"
format_stamp: "Formatted at 2026-09-25 21:56:21 on dist-test-slave-2g0f"
server_key: "d7fd328dfb703e3d8caf5b59a5956a96"
server_key_iv: "abb0566c35f30ce9f2206320192824e5"
server_key_version: "encryptionkey@0"
I20260925 21:56:21.304749 7548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "557be3a1439548f6ad2e04c9446645e0"
format_stamp: "Formatted at 2026-09-25 21:56:21 on dist-test-slave-2g0f"
server_key: "d7fd328dfb703e3d8caf5b59a5956a96"
server_key_iv: "abb0566c35f30ce9f2206320192824e5"
server_key_version: "encryptionkey@0"
I20260925 21:56:21.305935 7548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 21:56:21.306699 7562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:21.306880 7548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 21:56:21.306968 7548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "557be3a1439548f6ad2e04c9446645e0"
format_stamp: "Formatted at 2026-09-25 21:56:21 on dist-test-slave-2g0f"
server_key: "d7fd328dfb703e3d8caf5b59a5956a96"
server_key_iv: "abb0566c35f30ce9f2206320192824e5"
server_key_version: "encryptionkey@0"
I20260925 21:56:21.307039 7548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:21.325665 7548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:21.326006 7548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:21.326133 7548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:21.329790 7548 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:33607
I20260925 21:56:21.329839 7614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:33607 every 8 connection(s)
I20260925 21:56:21.330147 7548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:21.331108 7615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:21.333490 7615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0: Bootstrap starting.
I20260925 21:56:21.334028 7615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:21.334265 7615 log.cc:824] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:21.334524 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 7548
I20260925 21:56:21.334613 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:21.334828 2331 external_mini_cluster.cc:1428] Setting key fdd718a7d15a1417a68571738fbf40bc
I20260925 21:56:21.335089 7615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0: No bootstrap required, opened a new log
I20260925 21:56:21.337355 7615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557be3a1439548f6ad2e04c9446645e0" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33607 } }
I20260925 21:56:21.337502 7615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:21.337535 7615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557be3a1439548f6ad2e04c9446645e0, State: Initialized, Role: FOLLOWER
I20260925 21:56:21.337661 7615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557be3a1439548f6ad2e04c9446645e0" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33607 } }
I20260925 21:56:21.337733 7615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:21.337764 7615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:21.337811 7615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:21.338394 7615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557be3a1439548f6ad2e04c9446645e0" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33607 } }
I20260925 21:56:21.338496 7615 leader_election.cc:302] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 557be3a1439548f6ad2e04c9446645e0; no voters:
I20260925 21:56:21.338660 7615 leader_election.cc:288] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:21.338701 7620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:21.338807 7620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [term 1 LEADER]: Becoming Leader. State: Replica: 557be3a1439548f6ad2e04c9446645e0, State: Running, Role: LEADER
I20260925 21:56:21.338889 7620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557be3a1439548f6ad2e04c9446645e0" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33607 } }
I20260925 21:56:21.338924 7615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:21.339219 7620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 557be3a1439548f6ad2e04c9446645e0. Latest consensus state: current_term: 1 leader_uuid: "557be3a1439548f6ad2e04c9446645e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557be3a1439548f6ad2e04c9446645e0" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33607 } } }
I20260925 21:56:21.339285 7620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:21.339509 7622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "557be3a1439548f6ad2e04c9446645e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557be3a1439548f6ad2e04c9446645e0" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33607 } } }
I20260925 21:56:21.339635 7622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:21.339874 7625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:21.340368 7625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:21.340636 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:21.335331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52044 (local address 127.2.70.254:33607)
0925 21:56:21.335566 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:21.335574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:21.335602 (+ 28us) server_negotiation.cc:407] Connection header received
0925 21:56:21.336309 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:21.336317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:21.336383 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:21.336514 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:21.336716 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.337622 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.338468 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.339063 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.339179 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:21.339887 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:21.339891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:21.339891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:21.340105 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:21.340150 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:21.340165 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:21.340225 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:21.340342 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 21:56:21.340431 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260925 21:56:21.341511 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 21:56:21.342851 7625 catalog_manager.cc:1347] Generated new cluster ID: bc1d8fd93b1a419896c6a8088161705f
I20260925 21:56:21.342909 7625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:21.351902 7625 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:21.352619 7625 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:21.360005 7625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 557be3a1439548f6ad2e04c9446645e0: Generated new TSK 0
I20260925 21:56:21.360239 7625 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 21:56:21.392186 7645 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 21:56:21.392283 7645 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 21:56:21.392588 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:21.370399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52050 (local address 127.2.70.254:33607)
0925 21:56:21.370439 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:21.370441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:21.370490 (+ 49us) server_negotiation.cc:407] Connection header received
0925 21:56:21.370589 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:21.370590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:21.370599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:21.370659 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0925 21:56:21.370816 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.371356 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.372528 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.372755 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.372820 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:21.372996 (+ 176us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 21:56:21.392374 (+ 19378us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37229/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 21:56:21.392520 (+ 146us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52050: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37229/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 21:56:21.392639 7619 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52050: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37229/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 21:56:21.392720 7644 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:21.370301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:33607 (local address 127.0.0.1:52050)
0925 21:56:21.370431 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:21.370443 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:21.370510 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:21.370678 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:21.370679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:21.370681 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 21:56:21.370743 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:21.370746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.371432 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.371433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:21.372434 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:21.372438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.372838 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.372839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:21.372900 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:21.372906 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 21:56:21.392497 (+ 19591us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37229/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 21:56:21.392620 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.70.254:33607: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37229/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260925 21:56:21.393010 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 7548
2026-09-25T21:56:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (228 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 21:56:21.430154 2331 mini_oidc.cc:152] Starting JWKS server
I20260925 21:56:21.430871 2331 webserver.cc:460] Webserver started at https://127.0.0.1:37921/ using document root <none> and password file <none>
I20260925 21:56:21.430914 2331 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 21:56:21.431049 2331 webserver.cc:460] Webserver started at http://127.0.0.1:38249/ using document root <none> and password file <none>
2026-09-25T21:56:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:21Z Disabled control of system clock
I20260925 21:56:21.438683 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:44905
--webserver_interface=127.2.70.254
--webserver_port=0
--builtin_ntp_servers=127.2.70.212:37261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:44905
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37921/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 21:56:21.519661 7651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:21.519917 7651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:21.519949 7651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:21.521190 7651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:21.521247 7651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:21.521265 7651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:21.521282 7651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:21.521297 7651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 21:56:21.521313 7651 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 21:56:21.521332 7651 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37921/jwks.json
I20260925 21:56:21.522892 7651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:44905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:44905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37921/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:21.523172 7651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:21.523470 7651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:21.528733 7659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:21.528832 7651 server_base.cc:1034] running on GCE node
W20260925 21:56:21.528741 7656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:21.528746 7657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:21.529310 7651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:21.529589 7651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:21.530740 7651 hybrid_clock.cc:648] HybridClock initialized: now 1790373381530721 us; error 28 us; skew 500 ppm
I20260925 21:56:21.532016 7651 webserver.cc:460] Webserver started at http://127.2.70.254:46695/ using document root <none> and password file <none>
I20260925 21:56:21.532301 7651 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:21.532361 7651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:21.532475 7651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:21.533550 7651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "d17ed6aba6d84eee98238637ee2335c0"
format_stamp: "Formatted at 2026-09-25 21:56:21 on dist-test-slave-2g0f"
server_key: "23fa6f7068d10c2a34d2b261847f416d"
server_key_iv: "ab3631000bb4998fd21ea9512096dfbb"
server_key_version: "encryptionkey@0"
I20260925 21:56:21.533885 7651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "d17ed6aba6d84eee98238637ee2335c0"
format_stamp: "Formatted at 2026-09-25 21:56:21 on dist-test-slave-2g0f"
server_key: "23fa6f7068d10c2a34d2b261847f416d"
server_key_iv: "ab3631000bb4998fd21ea9512096dfbb"
server_key_version: "encryptionkey@0"
I20260925 21:56:21.535269 7651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:21.536126 7665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:21.536330 7651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:21.536425 7651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "d17ed6aba6d84eee98238637ee2335c0"
format_stamp: "Formatted at 2026-09-25 21:56:21 on dist-test-slave-2g0f"
server_key: "23fa6f7068d10c2a34d2b261847f416d"
server_key_iv: "ab3631000bb4998fd21ea9512096dfbb"
server_key_version: "encryptionkey@0"
I20260925 21:56:21.536487 7651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:21.550266 7651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:21.550587 7651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:21.550698 7651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:21.554764 7651 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:44905
I20260925 21:56:21.554803 7717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:44905 every 8 connection(s)
I20260925 21:56:21.555143 7651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:21.555855 7718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:21.558585 7718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0: Bootstrap starting.
I20260925 21:56:21.559191 7718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:21.559484 7718 log.cc:824] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:21.560253 7718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0: No bootstrap required, opened a new log
I20260925 21:56:21.561902 7718 raft_consensus.cc:348] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17ed6aba6d84eee98238637ee2335c0" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44905 } }
I20260925 21:56:21.562067 7718 raft_consensus.cc:374] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:21.562101 7718 raft_consensus.cc:729] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17ed6aba6d84eee98238637ee2335c0, State: Initialized, Role: FOLLOWER
I20260925 21:56:21.562258 7718 consensus_queue.cc:260] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17ed6aba6d84eee98238637ee2335c0" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44905 } }
I20260925 21:56:21.562327 7718 raft_consensus.cc:388] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:21.562357 7718 raft_consensus.cc:482] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:21.562408 7718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:21.563028 7718 raft_consensus.cc:504] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17ed6aba6d84eee98238637ee2335c0" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44905 } }
I20260925 21:56:21.563131 7718 leader_election.cc:302] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d17ed6aba6d84eee98238637ee2335c0; no voters:
I20260925 21:56:21.563172 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 7651
I20260925 21:56:21.563283 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:21.563355 7718 leader_election.cc:288] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:21.563403 7721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:21.563469 2331 external_mini_cluster.cc:1428] Setting key 09d0455a42fb26001ef8984bae556b47
I20260925 21:56:21.563520 7721 raft_consensus.cc:686] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [term 1 LEADER]: Becoming Leader. State: Replica: d17ed6aba6d84eee98238637ee2335c0, State: Running, Role: LEADER
I20260925 21:56:21.563617 7721 consensus_queue.cc:237] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17ed6aba6d84eee98238637ee2335c0" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44905 } }
I20260925 21:56:21.563638 7718 sys_catalog.cc:564] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:21.564432 7723 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d17ed6aba6d84eee98238637ee2335c0. Latest consensus state: current_term: 1 leader_uuid: "d17ed6aba6d84eee98238637ee2335c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17ed6aba6d84eee98238637ee2335c0" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44905 } } }
I20260925 21:56:21.564565 7723 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:21.564549 7722 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d17ed6aba6d84eee98238637ee2335c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17ed6aba6d84eee98238637ee2335c0" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44905 } } }
I20260925 21:56:21.564711 7722 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:21.565537 7729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:21.566074 7729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:21.567595 7729 catalog_manager.cc:1347] Generated new cluster ID: 2e41c041009f4dc9a89ac50f7874c44a
I20260925 21:56:21.567651 7729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:21.570025 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:21.564039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.2.70.254:44905)
0925 21:56:21.564361 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:21.564371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:21.564407 (+ 36us) server_negotiation.cc:407] Connection header received
0925 21:56:21.564544 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:21.564564 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:21.564652 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:21.565130 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:21.565244 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.566590 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.567701 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.568407 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.568533 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:21.569274 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:21.569279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:21.569279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:21.569496 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:21.569543 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:21.569560 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:21.569648 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:21.569705 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 21:56:21.569792 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":193,"threads_started":1}
I20260925 21:56:21.570990 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 21:56:21.580739 7729 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:21.581496 7729 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:21.588927 7729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d17ed6aba6d84eee98238637ee2335c0: Generated new TSK 0
I20260925 21:56:21.589156 7729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:56:21.599884 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:21.596004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.2.70.254:44905)
0925 21:56:21.596066 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:21.596068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:21.596107 (+ 39us) server_negotiation.cc:407] Connection header received
0925 21:56:21.596187 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:21.596188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:21.596202 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:21.596285 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0925 21:56:21.596420 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.597242 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.598021 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.598358 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.598427 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:21.598598 (+ 171us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 21:56:21.599628 (+ 1030us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37921/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 21:56:21.599770 (+ 142us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37921/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 21:56:21.599957 7725 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37921/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 21:56:21.601804 7747 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:21.595909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:44905 (local address 127.0.0.1:56590)
0925 21:56:21.596053 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:21.596063 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:21.596118 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:21.596295 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:21.596296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:21.596298 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 21:56:21.596359 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:21.596362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.597316 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.597317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:21.597947 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:21.597949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.598430 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.598431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:21.598478 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:21.598483 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 21:56:21.599755 (+ 1272us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37921/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 21:56:21.601705 (+ 1950us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.70.254:44905: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37921/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":1753,"thread_start_us":77,"threads_started":1}
I20260925 21:56:21.602088 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 7651
E20260925 21:56:21.612756 7748 webserver.cc:533] Webserver: sslize failed:
E20260925 21:56:21.612843 7748 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T21:56:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (209 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T21:56:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:21Z Disabled control of system clock
I20260925 21:56:21.647099 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:39701
--webserver_interface=127.2.70.254
--webserver_port=0
--builtin_ntp_servers=127.2.70.212:38273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:39701
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/testchainca.cert with env {}
W20260925 21:56:21.727155 7752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:21.727386 7752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:21.727409 7752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:21.728689 7752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:21.728744 7752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:21.728755 7752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:21.728766 7752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:21.728778 7752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 21:56:21.728790 7752 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 21:56:21.728801 7752 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 21:56:21.730428 7752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:39701
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:39701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:21.730682 7752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:21.730921 7752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:21.735754 7757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:21.735793 7758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:21.735803 7760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:21.735976 7752 server_base.cc:1034] running on GCE node
I20260925 21:56:21.736310 7752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:21.736598 7752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:21.737751 7752 hybrid_clock.cc:648] HybridClock initialized: now 1790373381737730 us; error 30 us; skew 500 ppm
I20260925 21:56:21.739121 7752 webserver.cc:460] Webserver started at http://127.2.70.254:43445/ using document root <none> and password file <none>
I20260925 21:56:21.739394 7752 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:21.739446 7752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:21.739567 7752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:21.740689 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "de1fb32768ea4298add5fd1e9aeb9ceb"
format_stamp: "Formatted at 2026-09-25 21:56:21 on dist-test-slave-2g0f"
server_key: "435dfa302f42a0b5dc4d70d11df5ca51"
server_key_iv: "49cd64529c9e12b0a0e51a6abb1efcea"
server_key_version: "encryptionkey@0"
I20260925 21:56:21.741047 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "de1fb32768ea4298add5fd1e9aeb9ceb"
format_stamp: "Formatted at 2026-09-25 21:56:21 on dist-test-slave-2g0f"
server_key: "435dfa302f42a0b5dc4d70d11df5ca51"
server_key_iv: "49cd64529c9e12b0a0e51a6abb1efcea"
server_key_version: "encryptionkey@0"
I20260925 21:56:21.742338 7752 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 21:56:21.743036 7766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:21.743218 7752 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 21:56:21.743301 7752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "de1fb32768ea4298add5fd1e9aeb9ceb"
format_stamp: "Formatted at 2026-09-25 21:56:21 on dist-test-slave-2g0f"
server_key: "435dfa302f42a0b5dc4d70d11df5ca51"
server_key_iv: "49cd64529c9e12b0a0e51a6abb1efcea"
server_key_version: "encryptionkey@0"
I20260925 21:56:21.743363 7752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:21.753861 7752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:21.754184 7752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:21.754325 7752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:21.758160 7752 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:39701
I20260925 21:56:21.758220 7818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:39701 every 8 connection(s)
I20260925 21:56:21.758562 7752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:21.759621 7819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:21.761304 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 7752
I20260925 21:56:21.761417 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:21.761579 2331 external_mini_cluster.cc:1428] Setting key 6977d01a05688a9ff6675afb37dfe07b
I20260925 21:56:21.762368 7819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb: Bootstrap starting.
I20260925 21:56:21.763085 7819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:21.763408 7819 log.cc:824] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:21.764209 7819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb: No bootstrap required, opened a new log
I20260925 21:56:21.766453 7819 raft_consensus.cc:348] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1fb32768ea4298add5fd1e9aeb9ceb" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39701 } }
I20260925 21:56:21.766639 7819 raft_consensus.cc:374] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:21.766676 7819 raft_consensus.cc:729] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1fb32768ea4298add5fd1e9aeb9ceb, State: Initialized, Role: FOLLOWER
I20260925 21:56:21.766806 7819 consensus_queue.cc:260] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1fb32768ea4298add5fd1e9aeb9ceb" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39701 } }
I20260925 21:56:21.766891 7819 raft_consensus.cc:388] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:21.766925 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:21.766983 7819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:21.767594 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1fb32768ea4298add5fd1e9aeb9ceb" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39701 } }
I20260925 21:56:21.767719 7819 leader_election.cc:302] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de1fb32768ea4298add5fd1e9aeb9ceb; no voters:
I20260925 21:56:21.767904 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:21.762151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40282 (local address 127.2.70.254:39701)
0925 21:56:21.762365 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:21.762374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:21.762406 (+ 32us) server_negotiation.cc:407] Connection header received
0925 21:56:21.762969 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:21.762979 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:21.763068 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:21.763228 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:21.763407 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.764737 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.765634 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.766304 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.766436 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:21.767163 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:21.767166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:21.767167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:21.767361 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:21.767396 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:21.767409 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:21.767463 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:21.767545 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 21:56:21.767633 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260925 21:56:21.767951 7819 leader_election.cc:288] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:21.767982 7824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:21.768085 7824 raft_consensus.cc:686] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [term 1 LEADER]: Becoming Leader. State: Replica: de1fb32768ea4298add5fd1e9aeb9ceb, State: Running, Role: LEADER
I20260925 21:56:21.768167 7824 consensus_queue.cc:237] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1fb32768ea4298add5fd1e9aeb9ceb" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39701 } }
I20260925 21:56:21.768220 7819 sys_catalog.cc:564] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:21.768502 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [sys.catalog]: SysCatalogTable state changed. Reason: New leader de1fb32768ea4298add5fd1e9aeb9ceb. Latest consensus state: current_term: 1 leader_uuid: "de1fb32768ea4298add5fd1e9aeb9ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1fb32768ea4298add5fd1e9aeb9ceb" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39701 } } }
I20260925 21:56:21.768514 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de1fb32768ea4298add5fd1e9aeb9ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1fb32768ea4298add5fd1e9aeb9ceb" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39701 } } }
I20260925 21:56:21.768591 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:21.768610 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:21.768942 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 21:56:21.769860 7839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:56:21.769917 7839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:56:21.769968 7830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:21.770115 7830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:21.771652 7830 catalog_manager.cc:1347] Generated new cluster ID: ecb53b166f7d48dc950921938c30b1d8
I20260925 21:56:21.771706 7830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:21.779259 7830 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:21.779791 7830 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:21.789518 7830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de1fb32768ea4298add5fd1e9aeb9ceb: Generated new TSK 0
I20260925 21:56:21.789716 7830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:56:21.839551 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:21.800411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.2.70.254:39701)
0925 21:56:21.800458 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:21.800460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:21.800528 (+ 68us) server_negotiation.cc:407] Connection header received
0925 21:56:21.800589 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:21.800590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:21.800603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:21.800695 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0925 21:56:21.800805 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.801617 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.802429 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.802740 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.802807 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:21.802959 (+ 152us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 21:56:21.839268 (+ 36309us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 21:56:21.839470 (+ 202us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40284: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 21:56:21.839637 7822 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40284: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 21:56:21.839627 7848 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:21.800309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:39701 (local address 127.0.0.1:40284)
0925 21:56:21.800456 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:21.800468 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:21.800527 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:21.800695 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:21.800696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:21.800698 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 21:56:21.800752 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:21.800755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.801707 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.801708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:21.802350 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:21.802352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:21.802816 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:21.802817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:21.802859 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:21.802864 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 21:56:21.839414 (+ 36550us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 21:56:21.839543 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.70.254:39701: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260925 21:56:21.840097 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 7752
2026-09-25T21:56:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (216 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:22 dist-test-slave-2g0f krb5kdc[7860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:22 dist-test-slave-2g0f krb5kdc[7860](info): set up 2 sockets
Sep 25 21:56:22 dist-test-slave-2g0f krb5kdc[7860](info): commencing operation
krb5kdc: starting...
W20260925 21:56:22.972012 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:56:23.092244 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:44803
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:44803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:23.174495 7873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:23.174722 7873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:23.174752 7873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:23.175988 7873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:23.176033 7873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:23.176051 7873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:23.176067 7873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:23.176083 7873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:23.177557 7873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:44803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:44803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:23.177804 7873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:23.178007 7873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:23.182746 7879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:23.182778 7881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:23.182780 7878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:23.182942 7873 server_base.cc:1034] running on GCE node
I20260925 21:56:23.183307 7873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:23.183580 7873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:23.184746 7873 hybrid_clock.cc:648] HybridClock initialized: now 1790373383184728 us; error 33 us; skew 500 ppm
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:23.187213 7873 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:56:23.187665 7873 webserver.cc:460] Webserver started at http://127.2.70.254:34817/ using document root <none> and password file <none>
I20260925 21:56:23.187919 7873 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:23.187971 7873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:23.188093 7873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:23.189109 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "e8965e40fc0649419ad41646351a90e4"
format_stamp: "Formatted at 2026-09-25 21:56:23 on dist-test-slave-2g0f"
server_key: "862bbe1308d9513ae81e10ae561f9a9a"
server_key_iv: "ebdc4bf73ac8690080dbd78f7805c375"
server_key_version: "encryptionkey@0"
I20260925 21:56:23.189435 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "e8965e40fc0649419ad41646351a90e4"
format_stamp: "Formatted at 2026-09-25 21:56:23 on dist-test-slave-2g0f"
server_key: "862bbe1308d9513ae81e10ae561f9a9a"
server_key_iv: "ebdc4bf73ac8690080dbd78f7805c375"
server_key_version: "encryptionkey@0"
I20260925 21:56:23.190677 7873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:23.191371 7888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:23.191525 7873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:23.191612 7873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "e8965e40fc0649419ad41646351a90e4"
format_stamp: "Formatted at 2026-09-25 21:56:23 on dist-test-slave-2g0f"
server_key: "862bbe1308d9513ae81e10ae561f9a9a"
server_key_iv: "ebdc4bf73ac8690080dbd78f7805c375"
server_key_version: "encryptionkey@0"
I20260925 21:56:23.191677 7873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:23.219053 7873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:23.221052 7873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:23.221204 7873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:23.225113 7873 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:44803
I20260925 21:56:23.225153 7940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:44803 every 8 connection(s)
I20260925 21:56:23.225490 7873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:23.226343 7941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:23.226934 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 7873
I20260925 21:56:23.227021 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:23.227198 2331 external_mini_cluster.cc:1428] Setting key ac01943922f37b10c2343a847c35b0b0
I20260925 21:56:23.229218 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4: Bootstrap starting.
I20260925 21:56:23.229929 7941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:23.230221 7941 log.cc:824] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:23.230926 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4: No bootstrap required, opened a new log
I20260925 21:56:23.233098 7941 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8965e40fc0649419ad41646351a90e4" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44803 } }
I20260925 21:56:23.233237 7941 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:23.233259 7941 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8965e40fc0649419ad41646351a90e4, State: Initialized, Role: FOLLOWER
I20260925 21:56:23.233367 7941 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8965e40fc0649419ad41646351a90e4" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44803 } }
I20260925 21:56:23.233441 7941 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:23.233469 7941 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:23.233513 7941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:23.234057 7941 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8965e40fc0649419ad41646351a90e4" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44803 } }
I20260925 21:56:23.234161 7941 leader_election.cc:302] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8965e40fc0649419ad41646351a90e4; no voters:
I20260925 21:56:23.234314 7941 leader_election.cc:288] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:23.234493 7947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:23.234586 7941 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:23.234620 7947 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [term 1 LEADER]: Becoming Leader. State: Replica: e8965e40fc0649419ad41646351a90e4, State: Running, Role: LEADER
I20260925 21:56:23.234794 7947 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8965e40fc0649419ad41646351a90e4" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44803 } }
I20260925 21:56:23.235379 7948 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8965e40fc0649419ad41646351a90e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8965e40fc0649419ad41646351a90e4" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44803 } } }
I20260925 21:56:23.235467 7948 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:23.235644 7955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:23.235368 7946 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8965e40fc0649419ad41646351a90e4. Latest consensus state: current_term: 1 leader_uuid: "e8965e40fc0649419ad41646351a90e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8965e40fc0649419ad41646351a90e4" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44803 } } }
I20260925 21:56:23.235785 7946 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:23.236261 7955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:23.237957 7955 catalog_manager.cc:1347] Generated new cluster ID: e4052b3e601f474c919b1c793b457a7a
I20260925 21:56:23.238005 7955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:23.238888 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:23.227704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.2.70.254:44803)
0925 21:56:23.227910 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:23.227919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:23.227953 (+ 34us) server_negotiation.cc:407] Connection header received
0925 21:56:23.228478 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:23.228488 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:23.228573 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:23.228730 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:23.229290 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.230575 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.231349 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.232007 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.232112 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:23.234359 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:23.234363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:23.234363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:23.234375 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:23.237067 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.237397 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.237400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.237401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.237503 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.237722 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.237723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.237723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.238015 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:23.238149 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:23.238543 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 21:56:23.238681 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
I20260925 21:56:23.256731 7955 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:23.257293 7955 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:23.265974 7955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8965e40fc0649419ad41646351a90e4: Generated new TSK 0
I20260925 21:56:23.266193 7955 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:56:23.302655 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:44803
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:23.384080 7967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:23.384286 7967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:23.384308 7967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:23.385459 7967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:23.385499 7967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:23.385524 7967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:23.386986 7967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:44803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:23.387220 7967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:23.387458 7967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:23.392326 7973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:23.392448 7975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:23.392407 7972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:23.392587 7967 server_base.cc:1034] running on GCE node
I20260925 21:56:23.392764 7967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:23.392953 7967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:23.394088 7967 hybrid_clock.cc:648] HybridClock initialized: now 1790373383394075 us; error 25 us; skew 500 ppm
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:23.396456 7967 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:56:23.396903 7967 webserver.cc:460] Webserver started at http://127.2.70.193:33035/ using document root <none> and password file <none>
I20260925 21:56:23.397078 7967 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:23.397116 7967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:23.397212 7967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:23.398228 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "6a84c9a24ab24205b65111b46b3fa326"
format_stamp: "Formatted at 2026-09-25 21:56:23 on dist-test-slave-2g0f"
server_key: "ebec5dedee7eefe202e1d521e67f818c"
server_key_iv: "47f615646679a229b6939055c905874d"
server_key_version: "encryptionkey@0"
I20260925 21:56:23.398504 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "6a84c9a24ab24205b65111b46b3fa326"
format_stamp: "Formatted at 2026-09-25 21:56:23 on dist-test-slave-2g0f"
server_key: "ebec5dedee7eefe202e1d521e67f818c"
server_key_iv: "47f615646679a229b6939055c905874d"
server_key_version: "encryptionkey@0"
I20260925 21:56:23.399694 7967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:23.400477 7982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:23.400643 7967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:56:23.400754 7967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "6a84c9a24ab24205b65111b46b3fa326"
format_stamp: "Formatted at 2026-09-25 21:56:23 on dist-test-slave-2g0f"
server_key: "ebec5dedee7eefe202e1d521e67f818c"
server_key_iv: "47f615646679a229b6939055c905874d"
server_key_version: "encryptionkey@0"
I20260925 21:56:23.400818 7967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:23.423159 7967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:23.424958 7967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:23.425097 7967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:23.425320 7967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:23.425604 7967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:23.425635 7967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:23.425674 7967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:23.425698 7967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:23.431308 7967 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:39593
I20260925 21:56:23.431361 8095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:39593 every 8 connection(s)
I20260925 21:56:23.431659 7967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:23.437911 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 7967
I20260925 21:56:23.438010 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:23.438220 2331 external_mini_cluster.cc:1428] Setting key c1c677c7c454c5c828cbff0bcc55aba6
I20260925 21:56:23.440714 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:23.432377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:38021 (local address 127.2.70.254:44803)
0925 21:56:23.432450 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:23.432452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:23.432601 (+ 149us) server_negotiation.cc:407] Connection header received
0925 21:56:23.433521 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:23.433522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:23.433533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:23.433592 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:23.434191 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.434727 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.435810 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.436028 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.436078 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:23.438000 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:23.438002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:23.438002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:23.438009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:23.439369 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.439724 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.439725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.439726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.439784 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.440104 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.440105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.440105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.440238 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:23.440302 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:23.440532 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 21:56:23.440611 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 21:56:23.440828 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:23.432324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:44803 (local address 127.2.70.193:38021)
0925 21:56:23.432475 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:23.432517 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:23.433428 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:23.433653 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:23.433658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:23.433898 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:23.434088 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:23.434092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.434794 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.434794 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:23.435743 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:23.435745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.436087 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.436088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:23.436194 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:23.436626 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:23.436634 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:23.437914 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:23.439445 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:23.439452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:23.439454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:23.439660 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:23.439889 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:23.439890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:23.439891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:23.440043 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:23.440295 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:23.440298 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:23.440418 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 21:56:23.440501 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 21:56:23.440624 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260925 21:56:23.441287 8096 heartbeater.cc:347] Connected to a master server at 127.2.70.254:44803
I20260925 21:56:23.441390 8096 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:23.441567 8096 heartbeater.cc:511] Master 127.2.70.254:44803 requested a full tablet report, sending...
I20260925 21:56:23.442078 7905 ts_manager.cc:194] Registered new tserver with Master: 6a84c9a24ab24205b65111b46b3fa326 (127.2.70.193:39593)
I20260925 21:56:23.442719 7905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:38021
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:56:23.463049 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:44803
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:23.542877 8101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:23.543103 8101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:23.543128 8101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:23.544327 8101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:23.544369 8101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:23.544395 8101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:23.545877 8101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:44803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:23.546080 8101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:23.546281 8101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:23.551362 8106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:23.551362 8107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:23.551431 8109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:23.551517 8101 server_base.cc:1034] running on GCE node
I20260925 21:56:23.551863 8101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:23.552090 8101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:23.553273 8101 hybrid_clock.cc:648] HybridClock initialized: now 1790373383553262 us; error 32 us; skew 500 ppm
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:23.555756 8101 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:56:23.556205 8101 webserver.cc:460] Webserver started at http://127.2.70.194:46351/ using document root <none> and password file <none>
I20260925 21:56:23.556418 8101 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:23.556468 8101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:23.556597 8101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:23.557682 8101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "4fdb5a613dc8473c8f04814302c23b28"
format_stamp: "Formatted at 2026-09-25 21:56:23 on dist-test-slave-2g0f"
server_key: "79dc8c1e41801e292e60d5c3802dc701"
server_key_iv: "0bf9604ad28c27a94531afb8e5c6ec39"
server_key_version: "encryptionkey@0"
I20260925 21:56:23.558012 8101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "4fdb5a613dc8473c8f04814302c23b28"
format_stamp: "Formatted at 2026-09-25 21:56:23 on dist-test-slave-2g0f"
server_key: "79dc8c1e41801e292e60d5c3802dc701"
server_key_iv: "0bf9604ad28c27a94531afb8e5c6ec39"
server_key_version: "encryptionkey@0"
I20260925 21:56:23.559262 8101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:23.559993 8116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:23.560180 8101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:23.560289 8101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "4fdb5a613dc8473c8f04814302c23b28"
format_stamp: "Formatted at 2026-09-25 21:56:23 on dist-test-slave-2g0f"
server_key: "79dc8c1e41801e292e60d5c3802dc701"
server_key_iv: "0bf9604ad28c27a94531afb8e5c6ec39"
server_key_version: "encryptionkey@0"
I20260925 21:56:23.560353 8101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:23.585944 8101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:23.588160 8101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:23.588305 8101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:23.588519 8101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:23.588821 8101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:23.588855 8101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:23.588884 8101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:23.588901 8101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:23.594524 8101 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:46871
I20260925 21:56:23.594566 8229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:46871 every 8 connection(s)
I20260925 21:56:23.594869 8101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:23.597885 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 8101
I20260925 21:56:23.597994 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:23.598166 2331 external_mini_cluster.cc:1428] Setting key 53f6a6346baa3403044affe9aa07ed2b
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:23.604599 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:23.595549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:57751 (local address 127.2.70.254:44803)
0925 21:56:23.595620 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:23.595622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:23.595799 (+ 177us) server_negotiation.cc:407] Connection header received
0925 21:56:23.596815 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:23.596816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:23.596830 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:23.596881 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:23.597536 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.598205 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.599345 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.599614 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.599683 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:23.601885 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:23.601886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:23.601887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:23.601895 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:23.603172 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.603593 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.603595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.603595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.603655 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.603900 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.603900 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.603901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.604020 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:23.604077 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:23.604339 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 21:56:23.604495 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 21:56:23.604713 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:23.595538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:44803 (local address 127.2.70.194:57751)
0925 21:56:23.595688 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:23.595723 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:23.596710 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:23.596952 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:23.596956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:23.597242 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:23.597447 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:23.597453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.598295 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.598296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:23.599264 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:23.599267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.599692 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.599692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:23.599804 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:23.600326 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:23.600337 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:23.601799 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:23.603247 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:23.603253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:23.603255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:23.603526 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:23.603708 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:23.603709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:23.603709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:23.603812 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:23.604075 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:23.604081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:23.604217 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 21:56:23.604299 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 21:56:23.604491 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260925 21:56:23.605100 8230 heartbeater.cc:347] Connected to a master server at 127.2.70.254:44803
I20260925 21:56:23.605188 8230 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:23.605360 8230 heartbeater.cc:511] Master 127.2.70.254:44803 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
I20260925 21:56:23.605726 7905 ts_manager.cc:194] Registered new tserver with Master: 4fdb5a613dc8473c8f04814302c23b28 (127.2.70.194:46871)
I20260925 21:56:23.606086 7905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:57751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:56:23.623006 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:44803
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:23.703305 8235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:23.703526 8235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:23.703559 8235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:23.704794 8235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:23.704838 8235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:23.704876 8235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:23.706391 8235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:44803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:23.706619 8235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:23.706827 8235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:23.711822 8243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:23.711928 8240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:23.712082 8241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:23.712080 8235 server_base.cc:1034] running on GCE node
I20260925 21:56:23.712347 8235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:23.712581 8235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:23.713729 8235 hybrid_clock.cc:648] HybridClock initialized: now 1790373383713724 us; error 39 us; skew 500 ppm
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:23.716202 8235 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:56:23.716640 8235 webserver.cc:460] Webserver started at http://127.2.70.195:43571/ using document root <none> and password file <none>
I20260925 21:56:23.716840 8235 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:23.716885 8235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:23.717003 8235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:23.718055 8235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "de0497a07796440ebbc74b40de80220b"
format_stamp: "Formatted at 2026-09-25 21:56:23 on dist-test-slave-2g0f"
server_key: "1f9fc1494ff0e53cf9fdf3d5b9012975"
server_key_iv: "b234a8e57d13b02bd6e4be5f30b2e7ec"
server_key_version: "encryptionkey@0"
I20260925 21:56:23.718402 8235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "de0497a07796440ebbc74b40de80220b"
format_stamp: "Formatted at 2026-09-25 21:56:23 on dist-test-slave-2g0f"
server_key: "1f9fc1494ff0e53cf9fdf3d5b9012975"
server_key_iv: "b234a8e57d13b02bd6e4be5f30b2e7ec"
server_key_version: "encryptionkey@0"
I20260925 21:56:23.719664 8235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 21:56:23.720436 8250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:23.720593 8235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:23.720674 8235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "de0497a07796440ebbc74b40de80220b"
format_stamp: "Formatted at 2026-09-25 21:56:23 on dist-test-slave-2g0f"
server_key: "1f9fc1494ff0e53cf9fdf3d5b9012975"
server_key_iv: "b234a8e57d13b02bd6e4be5f30b2e7ec"
server_key_version: "encryptionkey@0"
I20260925 21:56:23.720744 8235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:23.733846 8235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:23.736059 8235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:23.736197 8235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:23.736420 8235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:23.736713 8235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:23.736747 8235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:23.736775 8235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:23.736800 8235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:23.742386 8235 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:41953
I20260925 21:56:23.742431 8363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:41953 every 8 connection(s)
I20260925 21:56:23.742741 8235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:23.747697 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 8235
I20260925 21:56:23.747781 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:23.747951 2331 external_mini_cluster.cc:1428] Setting key 35b5eb6365dacf16d3d7d9ff932b035f
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:23.752096 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:23.743406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:41701 (local address 127.2.70.254:44803)
0925 21:56:23.743465 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:23.743467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:23.743677 (+ 210us) server_negotiation.cc:407] Connection header received
0925 21:56:23.744700 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:23.744702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:23.744715 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:23.744766 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:23.745376 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.745895 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.746864 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.747083 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.747138 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:23.749487 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:23.749488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:23.749489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:23.749497 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:23.750863 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.751277 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.751278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.751278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.751338 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.751529 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.751529 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.751530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.751641 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:23.751691 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:23.751949 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 21:56:23.752022 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 21:56:23.752287 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:23.743379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:44803 (local address 127.2.70.195:41701)
0925 21:56:23.743555 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:23.743600 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:23.744595 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:23.744837 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:23.744841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:23.745095 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:23.745290 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:23.745294 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.745973 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.745974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:23.746786 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:23.746788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.747151 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.747152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:23.747274 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:23.747870 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:23.747880 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:23.749405 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:23.750942 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:23.750949 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:23.750951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:23.751178 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:23.751396 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:23.751397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:23.751397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:23.751487 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:23.751685 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:23.751687 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:23.751807 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 21:56:23.751930 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 21:56:23.752063 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260925 21:56:23.752709 8364 heartbeater.cc:347] Connected to a master server at 127.2.70.254:44803
I20260925 21:56:23.752801 8364 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:23.752949 8364 heartbeater.cc:511] Master 127.2.70.254:44803 requested a full tablet report, sending...
I20260925 21:56:23.753283 7905 ts_manager.cc:194] Registered new tserver with Master: de0497a07796440ebbc74b40de80220b (127.2.70.195:41953)
I20260925 21:56:23.753649 7905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:41701
I20260925 21:56:23.759740 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/test-admin.keytab.
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:23.777066 2331 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 21:56:23.778688 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:23.777886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.2.70.193:39593)
0925 21:56:23.778025 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:23.778030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:23.778039 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:23.778065 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:23.778067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:23.778080 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:23.778243 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:23.778607 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44446: BlockingRecv error: recv got EOF from 127.0.0.1:44446 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
W20260925 21:56:23.778857 2331 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/krb5cc' of type FILE
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:23.779873 2331 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:23.780727 2331 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.193@KRBTEST.COM
I20260925 21:56:23.788447 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:23.781311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.2.70.193:39593)
0925 21:56:23.781363 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:23.781365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:23.781373 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:23.781443 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:23.781444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:23.781454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:23.781499 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:23.781890 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.782625 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.783415 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.783751 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.783810 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:23.785917 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:23.785929 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:23.785930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:23.785936 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:23.787281 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.787635 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.787638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.787639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.787694 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.787894 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.787894 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.787895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.788040 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:23.788184 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:23.788253 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 21:56:23.788333 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 21:56:23.790203 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:23.789707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.2.70.193:39593)
0925 21:56:23.789765 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:23.789767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:23.789779 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:56:23.789845 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:23.789845 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:23.789855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:23.789899 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:23.790153 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44476: BlockingRecv error: recv got EOF from 127.0.0.1:44476 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 21:56:23.790239 2331 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/krb5cc' of type FILE
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:23.791208 2331 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:23.792026 2331 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.193@KRBTEST.COM
I20260925 21:56:23.799181 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:23.792638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.2.70.193:39593)
0925 21:56:23.792689 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:23.792691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:23.792714 (+ 23us) server_negotiation.cc:407] Connection header received
0925 21:56:23.792731 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:23.792731 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:23.792739 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:23.792776 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:23.793116 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.793627 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.794323 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.794521 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.794575 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:23.796687 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:23.796688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:23.796689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:23.796695 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:23.797970 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.798357 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.798358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.798358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.798413 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.798608 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.798610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.798610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.798721 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:23.798821 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:23.799008 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 21:56:23.799110 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 21:56:23.800747 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:23.800239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44490 (local address 127.2.70.193:39593)
0925 21:56:23.800277 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:23.800279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:23.800327 (+ 48us) server_negotiation.cc:407] Connection header received
0925 21:56:23.800384 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:23.800385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:23.800394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:23.800441 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:23.800702 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44490: BlockingRecv error: recv got EOF from 127.0.0.1:44490 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 21:56:23.800820 2331 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790373357410105-2331-0/krb5kdc/krb5cc' of type FILE
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:23.801795 2331 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:23.802582 2331 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.193@KRBTEST.COM
I20260925 21:56:23.809118 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:23.803026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.2.70.193:39593)
0925 21:56:23.803061 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:23.803063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:23.803121 (+ 58us) server_negotiation.cc:407] Connection header received
0925 21:56:23.803174 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:23.803175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:23.803185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:23.803226 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:23.803554 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.804115 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.804838 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:23.805025 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:23.805075 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:23.806940 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:23.806941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:23.806941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:23.806947 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:23.808178 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.808470 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.808471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.808472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.808544 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:23.808753 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:23.808754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:23.808754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:23.808837 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:23.808899 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:23.808991 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 21:56:23.809055 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 21:56:23.809648 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 7967
I20260925 21:56:23.814061 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 8101
I20260925 21:56:23.818421 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 8235
I20260925 21:56:23.821848 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 7873
2026-09-25T21:56:23Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1808 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[8384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[8384](info): set up 2 sockets
Sep 25 21:56:23 dist-test-slave-2g0f krb5kdc[8384](info): commencing operation
krb5kdc: starting...
W20260925 21:56:24.775949 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:24 dist-test-slave-2g0f krb5kdc[8384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.254.keytab.
Entry for principal oryx/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.254.keytab.
I20260925 21:56:24.895008 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:43873
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.254.keytab
--principal=oryx/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:43873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:24.974427 8397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:24.974637 8397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:24.974664 8397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:24.975783 8397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:24.975819 8397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:24.975857 8397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:24.975868 8397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:24.975879 8397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:24.977275 8397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:43873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.254.keytab
--principal=oryx/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:43873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:24.977468 8397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:24.977631 8397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:24.982419 8402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:24.982375 8403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:24.982375 8405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:24.982425 8397 server_base.cc:1034] running on GCE node
I20260925 21:56:24.982877 8397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:24.983179 8397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:24.984339 8397 hybrid_clock.cc:648] HybridClock initialized: now 1790373384984328 us; error 36 us; skew 500 ppm
Sep 25 21:56:24 dist-test-slave-2g0f krb5kdc[8384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373384, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:24.986873 8397 init.cc:377] Logged in from keytab as oryx/127.2.70.254@KRBTEST.COM (short username oryx)
I20260925 21:56:24.987397 8397 webserver.cc:460] Webserver started at http://127.2.70.254:38675/ using document root <none> and password file <none>
I20260925 21:56:24.987607 8397 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:24.987660 8397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:24.987797 8397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:24.988911 8397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "1de8b50ece3044268f93c293784dc39b"
format_stamp: "Formatted at 2026-09-25 21:56:24 on dist-test-slave-2g0f"
server_key: "e373ace5eee97869ebb9928a0c703b6c"
server_key_iv: "005ca883184f8c397233b73171fa0dbd"
server_key_version: "encryptionkey@0"
I20260925 21:56:24.989288 8397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "1de8b50ece3044268f93c293784dc39b"
format_stamp: "Formatted at 2026-09-25 21:56:24 on dist-test-slave-2g0f"
server_key: "e373ace5eee97869ebb9928a0c703b6c"
server_key_iv: "005ca883184f8c397233b73171fa0dbd"
server_key_version: "encryptionkey@0"
I20260925 21:56:24.990670 8397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:24.991508 8412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:24.991690 8397 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:24.991783 8397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "1de8b50ece3044268f93c293784dc39b"
format_stamp: "Formatted at 2026-09-25 21:56:24 on dist-test-slave-2g0f"
server_key: "e373ace5eee97869ebb9928a0c703b6c"
server_key_iv: "005ca883184f8c397233b73171fa0dbd"
server_key_version: "encryptionkey@0"
I20260925 21:56:24.991878 8397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:25.010270 8397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:25.012591 8397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:25.012744 8397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:25.016508 8397 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:43873
I20260925 21:56:25.016543 8464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:43873 every 8 connection(s)
I20260925 21:56:25.016878 8397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:25.017831 8465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:25.019805 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 8397
I20260925 21:56:25.019934 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:25.020139 2331 external_mini_cluster.cc:1428] Setting key c95986cfc4c35243c193b8a0265a1146
I20260925 21:56:25.020372 8465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b: Bootstrap starting.
I20260925 21:56:25.021039 8465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:25.021309 8465 log.cc:824] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:25.022110 8465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b: No bootstrap required, opened a new log
I20260925 21:56:25.024115 8465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } }
I20260925 21:56:25.024295 8465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:25.024324 8465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de8b50ece3044268f93c293784dc39b, State: Initialized, Role: FOLLOWER
I20260925 21:56:25.024456 8465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } }
I20260925 21:56:25.024550 8465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:25.024574 8465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:25.024619 8465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:25.025394 8465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } }
I20260925 21:56:25.025512 8465 leader_election.cc:302] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1de8b50ece3044268f93c293784dc39b; no voters:
I20260925 21:56:25.025708 8465 leader_election.cc:288] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:25.025743 8470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:25.025885 8470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 1 LEADER]: Becoming Leader. State: Replica: 1de8b50ece3044268f93c293784dc39b, State: Running, Role: LEADER
I20260925 21:56:25.026010 8465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:25.026069 8470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } }
I20260925 21:56:25.026417 8470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1de8b50ece3044268f93c293784dc39b. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } }
I20260925 21:56:25.026424 8472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } }
I20260925 21:56:25.026573 8472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:25.026764 8470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:25.027077 8477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
I20260925 21:56:25.027560 8477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:25.030036 8477 catalog_manager.cc:1347] Generated new cluster ID: b18e6ffcbfb144a8a82a34aecb3db065
I20260925 21:56:25.030133 8477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:25.031241 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.020671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.2.70.254:43873)
0925 21:56:25.020835 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.020845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.020872 (+ 27us) server_negotiation.cc:407] Connection header received
0925 21:56:25.021354 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.021364 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.021436 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.021569 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.022278 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.023159 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.024205 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.024741 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.024821 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.027710 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.027715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.027715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.027729 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.029517 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.029903 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.029906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.029907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.030031 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.030251 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.030252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.030252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.030581 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.030753 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.030854 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.031013 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260925 21:56:25.049715 8477 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:25.050350 8477 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:25.059799 8477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b: Generated new TSK 0
I20260925 21:56:25.060048 8477 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab.
Entry for principal oryx/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab.
I20260925 21:56:25.092938 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:43873
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab
--principal=oryx/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:25.175747 8491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:25.176002 8491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:25.176026 8491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:25.177210 8491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:25.177248 8491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:25.177275 8491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:25.178735 8491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab
--principal=oryx/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:43873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:25.178960 8491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:25.179143 8491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:25.183909 8497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:25.183923 8499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:25.184011 8491 server_base.cc:1034] running on GCE node
W20260925 21:56:25.183909 8496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:25.184303 8491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:25.184545 8491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:25.185698 8491 hybrid_clock.cc:648] HybridClock initialized: now 1790373385185680 us; error 32 us; skew 500 ppm
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:25.188217 8491 init.cc:377] Logged in from keytab as oryx/127.2.70.193@KRBTEST.COM (short username oryx)
I20260925 21:56:25.188647 8491 webserver.cc:460] Webserver started at http://127.2.70.193:42725/ using document root <none> and password file <none>
I20260925 21:56:25.188858 8491 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:25.188905 8491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:25.189024 8491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:25.190037 8491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "0882610850c9425ea77ae50c3a43dfbb"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "ddf7d33858db8eb4ecd9fdc3ee04ea22"
server_key_iv: "f25dbbedeb5703c79dc4107f6c95512c"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.190362 8491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "0882610850c9425ea77ae50c3a43dfbb"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "ddf7d33858db8eb4ecd9fdc3ee04ea22"
server_key_iv: "f25dbbedeb5703c79dc4107f6c95512c"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.191583 8491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:25.192302 8506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.192479 8491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.192566 8491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "0882610850c9425ea77ae50c3a43dfbb"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "ddf7d33858db8eb4ecd9fdc3ee04ea22"
server_key_iv: "f25dbbedeb5703c79dc4107f6c95512c"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.192636 8491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:25.209487 8491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:25.211632 8491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:25.211768 8491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:25.212021 8491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:25.212332 8491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:25.212364 8491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.212392 8491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:25.212415 8491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.217839 8491 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:43971
I20260925 21:56:25.217912 8619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:43971 every 8 connection(s)
I20260925 21:56:25.218202 8491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
I20260925 21:56:25.226908 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.218869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:36691 (local address 127.2.70.254:43873)
0925 21:56:25.218912 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.218914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.219111 (+ 197us) server_negotiation.cc:407] Connection header received
0925 21:56:25.219559 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.219560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.219571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.219616 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.220595 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.221108 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.222101 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.222307 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.222356 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.224468 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.224470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.224470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.224477 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.225658 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.226073 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.226074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.226074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.226128 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.226313 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.226314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.226314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.226411 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.226454 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.226724 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.226793 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 21:56:25.227102 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.218849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.193:36691)
0925 21:56:25.218992 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.219027 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.219485 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.219682 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.219685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.220279 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.220491 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.220495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.221194 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.221195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.222000 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.222003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.222366 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.222366 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.222459 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.222932 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.222940 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.224374 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.225749 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.225757 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.225759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.225978 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.226179 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.226180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.226180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.226272 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.226452 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.226455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.226574 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.226690 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.226839 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260925 21:56:25.227559 8620 heartbeater.cc:347] Connected to a master server at 127.2.70.254:43873
I20260925 21:56:25.227654 8620 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:25.227906 8620 heartbeater.cc:511] Master 127.2.70.254:43873 requested a full tablet report, sending...
I20260925 21:56:25.227931 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 8491
I20260925 21:56:25.228037 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:25.228259 2331 external_mini_cluster.cc:1428] Setting key f7ddf91272f1a49ec6f3d7e9c42ec008
I20260925 21:56:25.228482 8429 ts_manager.cc:194] Registered new tserver with Master: 0882610850c9425ea77ae50c3a43dfbb (127.2.70.193:43971)
I20260925 21:56:25.229265 8429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:36691
WARNING: no policy specified for oryx/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab.
Entry for principal oryx/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab.
I20260925 21:56:25.254904 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:43873
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab
--principal=oryx/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:25.335995 8625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:25.336212 8625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:25.336236 8625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:25.337443 8625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:25.337486 8625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:25.337512 8625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:25.338939 8625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab
--principal=oryx/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:43873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:25.339165 8625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:25.339383 8625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:25.344307 8631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:25.344307 8630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:25.344501 8625 server_base.cc:1034] running on GCE node
W20260925 21:56:25.344385 8633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:25.344740 8625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:25.345033 8625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:25.346199 8625 hybrid_clock.cc:648] HybridClock initialized: now 1790373385346173 us; error 39 us; skew 500 ppm
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:25.348770 8625 init.cc:377] Logged in from keytab as oryx/127.2.70.194@KRBTEST.COM (short username oryx)
I20260925 21:56:25.349325 8625 webserver.cc:460] Webserver started at http://127.2.70.194:34419/ using document root <none> and password file <none>
I20260925 21:56:25.349524 8625 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:25.349568 8625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:25.349676 8625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:25.350703 8625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "0be7ac9f2c9d4849b2f19f16a277448d"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "3c3628114ffd0133460cc8c9ec1455a3"
server_key_iv: "8b8a335468d20924d87e42f701c415c3"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.351027 8625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "0be7ac9f2c9d4849b2f19f16a277448d"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "3c3628114ffd0133460cc8c9ec1455a3"
server_key_iv: "8b8a335468d20924d87e42f701c415c3"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.352295 8625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 21:56:25.353044 8640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.353210 8625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:25.353291 8625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "0be7ac9f2c9d4849b2f19f16a277448d"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "3c3628114ffd0133460cc8c9ec1455a3"
server_key_iv: "8b8a335468d20924d87e42f701c415c3"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.353353 8625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:25.371019 8625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:25.372915 8625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:25.373047 8625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:25.373296 8625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:25.373574 8625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:25.373605 8625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.373625 8625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:25.373644 8625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.379099 8625 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:35501
I20260925 21:56:25.379148 8753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:35501 every 8 connection(s)
I20260925 21:56:25.379433 8625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:25.379662 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 8625
I20260925 21:56:25.379769 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:25.379981 2331 external_mini_cluster.cc:1428] Setting key 161c023b65d72b196c26e2e3c63e7f89
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.70.195@KRBTEST.COM; defaulting to no policy
I20260925 21:56:25.389556 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.380234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:52171 (local address 127.2.70.254:43873)
0925 21:56:25.380286 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.380288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.380467 (+ 179us) server_negotiation.cc:407] Connection header received
0925 21:56:25.381032 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.381033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.381042 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.381079 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.382268 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.383007 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.384397 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.384598 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.384653 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.387018 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.387020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.387020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.387028 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.388277 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.388653 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.388654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.388655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.388707 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.388929 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.388930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.388930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.389039 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.389094 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.389375 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.389453 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 21:56:25.389730 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.380212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.194:52171)
0925 21:56:25.380356 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.380396 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.380934 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.381152 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.381157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.381882 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.382170 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.382177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.383097 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.383097 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.384306 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.384309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.384670 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.384672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.384823 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.385327 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.385337 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.386923 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.388351 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.388358 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.388360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.388589 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.388760 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.388761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.388761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.388876 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.389093 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.389099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.389231 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.389351 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.389521 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260925 21:56:25.390141 8754 heartbeater.cc:347] Connected to a master server at 127.2.70.254:43873
I20260925 21:56:25.390254 8754 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:25.390444 8754 heartbeater.cc:511] Master 127.2.70.254:43873 requested a full tablet report, sending...
I20260925 21:56:25.390803 8429 ts_manager.cc:194] Registered new tserver with Master: 0be7ac9f2c9d4849b2f19f16a277448d (127.2.70.194:35501)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.195@KRBTEST.COM" created.
I20260925 21:56:25.391292 8429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:52171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab.
Entry for principal oryx/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab.
I20260925 21:56:25.406319 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:43873
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab
--principal=oryx/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:25.486279 8759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:25.486480 8759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:25.486510 8759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:25.487649 8759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:25.487689 8759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:25.487715 8759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:25.489116 8759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab
--principal=oryx/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:43873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:25.489295 8759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:25.489472 8759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:25.494239 8765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:25.494269 8767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:25.494234 8764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:25.494266 8759 server_base.cc:1034] running on GCE node
I20260925 21:56:25.494666 8759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:25.494885 8759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:25.496022 8759 hybrid_clock.cc:648] HybridClock initialized: now 1790373385496010 us; error 26 us; skew 500 ppm
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:25.498387 8759 init.cc:377] Logged in from keytab as oryx/127.2.70.195@KRBTEST.COM (short username oryx)
I20260925 21:56:25.498770 8759 webserver.cc:460] Webserver started at http://127.2.70.195:34579/ using document root <none> and password file <none>
I20260925 21:56:25.498956 8759 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:25.499003 8759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:25.499094 8759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:25.500128 8759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "d15c2de96cec44be9ed402dbaead6aba"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "6fb36d870c9f4fa49dc9414b8f4b86a1"
server_key_iv: "c403804d50555235f2c65d08e6a62f5f"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.500408 8759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "d15c2de96cec44be9ed402dbaead6aba"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "6fb36d870c9f4fa49dc9414b8f4b86a1"
server_key_iv: "c403804d50555235f2c65d08e6a62f5f"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.501611 8759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:25.502396 8774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.502576 8759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:25.502657 8759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "d15c2de96cec44be9ed402dbaead6aba"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "6fb36d870c9f4fa49dc9414b8f4b86a1"
server_key_iv: "c403804d50555235f2c65d08e6a62f5f"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.502714 8759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:25.518805 8759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:25.520967 8759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:25.521098 8759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:25.521309 8759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:25.521600 8759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:25.521632 8759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.521660 8759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:25.521680 8759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.527333 8759 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:39491
I20260925 21:56:25.527374 8887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:39491 every 8 connection(s)
I20260925 21:56:25.527720 8759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:25.531196 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 8759
I20260925 21:56:25.531301 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:25.531488 2331 external_mini_cluster.cc:1428] Setting key 459947ad26b5658eb7e36b61a561ac8b
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
I20260925 21:56:25.537544 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.528456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:54613 (local address 127.2.70.254:43873)
0925 21:56:25.528523 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.528525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.528708 (+ 183us) server_negotiation.cc:407] Connection header received
0925 21:56:25.529240 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.529242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.529254 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.529304 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.530229 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.530746 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.532132 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.532402 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.532470 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.534795 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.534797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.534797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.534804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.536227 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.536632 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.536633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.536633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.536702 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.536933 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.536934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.536935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.537109 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.537175 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.537396 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.537470 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 21:56:25.537725 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.528420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.195:54613)
0925 21:56:25.528591 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.528629 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.529137 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.529375 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.529379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.529946 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.530139 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.530143 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.530823 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.530825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.532051 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.532054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.532482 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.532483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.532588 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.533193 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.533202 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.534694 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.536299 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.536305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.536306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.536545 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.536763 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.536763 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.536764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.536855 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.537170 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.537173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.537292 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.537374 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.537513 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260925 21:56:25.538184 8888 heartbeater.cc:347] Connected to a master server at 127.2.70.254:43873
I20260925 21:56:25.538280 8888 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:25.538452 8888 heartbeater.cc:511] Master 127.2.70.254:43873 requested a full tablet report, sending...
I20260925 21:56:25.538863 8429 ts_manager.cc:194] Registered new tserver with Master: d15c2de96cec44be9ed402dbaead6aba (127.2.70.195:39491)
I20260925 21:56:25.539242 8429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:54613
I20260925 21:56:25.543279 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.70.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.253.keytab.
Entry for principal oryx/127.2.70.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.253.keytab.
I20260925 21:56:25.570439 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.253:43233
--webserver_interface=127.2.70.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.253.keytab
--principal=oryx/127.2.70.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:43873,127.2.70.253:43233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:25.651191 8893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:25.651417 8893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:25.651440 8893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:25.652596 8893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:25.652642 8893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:25.652657 8893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:25.652670 8893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:25.652683 8893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:25.654162 8893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:43873,127.2.70.253:43233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.253.keytab
--principal=oryx/127.2.70.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.253:43233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.70.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
W20260925 21:56:25.654281 8893 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.70.254:43873,127.2.70.253:43233'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 21:56:25.654398 8893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:25.654587 8893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:56:25.659562 8893 server_base.cc:1034] running on GCE node
W20260925 21:56:25.659660 8901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:25.659577 8899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:25.659744 8898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:25.660101 8893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:25.660393 8893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:25.661532 8893 hybrid_clock.cc:648] HybridClock initialized: now 1790373385661517 us; error 33 us; skew 500 ppm
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:25.664198 8893 init.cc:377] Logged in from keytab as oryx/127.2.70.253@KRBTEST.COM (short username oryx)
I20260925 21:56:25.664657 8893 webserver.cc:460] Webserver started at http://127.2.70.253:46465/ using document root <none> and password file <none>
I20260925 21:56:25.664880 8893 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:25.664925 8893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:25.665020 8893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:25.666046 8893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data/instance:
uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "aa8a4fdabe05c44e3de791521396ea18"
server_key_iv: "20dab6a18dc1bb577a4f0878456ab67c"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.666347 8893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/wal/instance:
uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "aa8a4fdabe05c44e3de791521396ea18"
server_key_iv: "20dab6a18dc1bb577a4f0878456ab67c"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.667608 8893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 21:56:25.668435 8908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.668648 8893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:56:25.668744 8893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/wal
uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "aa8a4fdabe05c44e3de791521396ea18"
server_key_iv: "20dab6a18dc1bb577a4f0878456ab67c"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.668812 8893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:25.681869 8893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:25.683732 8893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:25.683893 8893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:25.687347 8893 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.253:43233
I20260925 21:56:25.687392 8960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.253:43233 every 8 connection(s)
I20260925 21:56:25.687706 8893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
I20260925 21:56:25.688694 8961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:25.690281 8961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } has no permanent_uuid. Determining permanent_uuid...
I20260925 21:56:25.695271 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 8893
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.253@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
I20260925 21:56:25.699975 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.691061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.2.70.254:43873)
0925 21:56:25.691157 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.691160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.691376 (+ 216us) server_negotiation.cc:407] Connection header received
0925 21:56:25.691915 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.691917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.691930 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.692064 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.692835 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.693384 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.694374 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.694604 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.694665 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.697074 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.697075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.697075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.697084 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.698495 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.698850 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.698851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.698851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.698906 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.699156 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.699157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.699157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.699345 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.699398 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.699773 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.699897 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 21:56:25.700157 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.691045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:50522)
0925 21:56:25.691261 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.691298 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.691763 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.692201 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.692225 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.692513 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.692735 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.692740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.693450 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.693451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.694282 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.694285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.694695 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.694696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.694835 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.695331 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.695339 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.696992 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.698569 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.698574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.698576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.698776 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.698960 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.698961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.698961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.699095 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.699405 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.699411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.699587 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.699709 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.699926 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":116,"threads_started":1}
I20260925 21:56:25.700739 8961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.253@KRBTEST.COM
I20260925 21:56:25.704141 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.695717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52482 (local address 127.2.70.253:43233)
0925 21:56:25.695958 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.695962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.695978 (+ 16us) server_negotiation.cc:407] Connection header received
0925 21:56:25.696020 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.696022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.696042 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.696173 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.696751 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.697609 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.698353 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.698754 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.698802 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.701416 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.701425 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.701425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.701437 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.702897 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.703280 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.703283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.703284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.703379 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.703557 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.703558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.703558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.703714 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.703885 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.704012 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.704069 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":79,"threads_started":1}
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.253@KRBTEST.COM for oryx/127.2.70.253@KRBTEST.COM
I20260925 21:56:25.707298 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.700996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43233 (local address 127.0.0.1:52490)
0925 21:56:25.701171 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.701181 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.701212 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.701326 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.701327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.701460 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.701506 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.701508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.702102 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.702103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.702571 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.702573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.702884 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.702885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.702943 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.703281 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.703285 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.704755 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.706556 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.706557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.706558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.706710 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.706851 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.706851 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.706852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.706895 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.707068 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.707069 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.707103 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.707149 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.707218 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260925 21:56:25.707372 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.701045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.2.70.253:43233)
0925 21:56:25.701195 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.701196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.701217 (+ 21us) server_negotiation.cc:407] Connection header received
0925 21:56:25.701264 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.701265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.701274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.701326 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.701566 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.702043 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.702633 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.702832 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.702883 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.704825 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.704826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.704826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.704832 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.706491 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.706766 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.706766 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.706766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.706810 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.706928 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.706928 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.706928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.707026 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.707068 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.707165 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.707258 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260925 21:56:25.708568 8961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: Bootstrap starting.
I20260925 21:56:25.709107 8961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:25.709313 8961 log.cc:824] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:25.709916 8961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: No bootstrap required, opened a new log
I20260925 21:56:25.711218 8961 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } }
I20260925 21:56:25.711331 8961 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:25.711354 8961 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0b5e1ad65054f35a797c5a6b0e45ed6, State: Initialized, Role: FOLLOWER
I20260925 21:56:25.711448 8961 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } }
I20260925 21:56:25.711670 8968 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } } }
I20260925 21:56:25.711720 8961 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:25.711758 8968 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 21:56:25.713261 8979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:56:25.713343 8979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:56:25.715008 8893 master_runner.cc:428] Detected that this master f0b5e1ad65054f35a797c5a6b0e45ed6 is joining an existing cluster
I20260925 21:56:25.715058 8893 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.70.253:43233
I20260925 21:56:25.721872 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.716108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.2.70.254:43873)
0925 21:56:25.716168 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.716170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.716289 (+ 119us) server_negotiation.cc:407] Connection header received
0925 21:56:25.716371 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.716373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.716384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.716445 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.716776 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.717282 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.718039 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.718248 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.718308 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.719304 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.719305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.719306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.719313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.720750 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.721085 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.721086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.721087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.721154 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.721354 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.721355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.721356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.721500 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.721566 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.721705 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.721782 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 21:56:25.721884 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.716050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:50538)
0925 21:56:25.716233 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.716244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.716302 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.716448 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.716449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.716626 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.716697 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.716698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.717357 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.717358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.717952 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.717954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.718330 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.718331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.718388 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.718885 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.718889 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.719217 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.720820 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.720821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.720821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.721020 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.721215 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.721216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.721217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.721286 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.721562 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.721563 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.721604 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.721675 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.721770 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260925 21:56:25.726675 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.716085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43233 (local address 127.0.0.1:52498)
0925 21:56:25.719933 (+ 3848us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.719944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.720005 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.720352 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.720353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.720910 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.720967 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.720969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.721756 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.721757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.722509 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.722512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.723012 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.723013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.723067 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.723487 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.723491 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.723868 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.725608 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.725609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.725609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.725825 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.726075 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.726076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.726076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.726132 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.726346 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.726347 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.726383 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.726444 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.726546 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3831,"thread_start_us":72,"threads_started":1}
I20260925 21:56:25.726680 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.716239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.2.70.253:43233)
0925 21:56:25.716276 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.716278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.720012 (+ 3734us) server_negotiation.cc:407] Connection header received
0925 21:56:25.720264 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.720266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.720277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.720342 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.721046 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.721674 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.722585 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.722907 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.722970 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.723939 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.723940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.723941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.723948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.725532 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.725957 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.725958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.725958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.726017 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.726186 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.726187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.726187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.726291 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.726349 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.726480 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.726600 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":21}
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373384, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.254@KRBTEST.COM for oryx/127.2.70.253@KRBTEST.COM
I20260925 21:56:25.732402 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.724293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43233 (local address 127.0.0.1:52514)
0925 21:56:25.724435 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.724484 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.724568 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.724724 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.724730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.725043 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.725094 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.725096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.725770 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.725771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.726706 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.726708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.727434 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.727435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.727520 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.727999 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.728005 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.729457 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.731256 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.731262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.731264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.731456 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.731651 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.731652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.731652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.731716 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.732055 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.732062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.732121 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.732204 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.732312 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260925 21:56:25.732441 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.724336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.2.70.253:43233)
0925 21:56:25.724377 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.724379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.724554 (+ 175us) server_negotiation.cc:407] Connection header received
0925 21:56:25.724650 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.724651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.724660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.724715 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.725177 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.725693 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.727020 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.727301 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.727368 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.729548 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.729549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.729550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.729556 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.731150 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.731519 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.731520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.731521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.731573 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.731776 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.731777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.731777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.731925 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.731994 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.732252 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.732343 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 21:56:25.732954 8429 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.70.253:43233: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6"
member_type: NON_VOTER
last_known_addr {
host: "127.2.70.253"
port: 43233
}
attrs {
promote: true
}
}
dest_uuid: "1de8b50ece3044268f93c293784dc39b"
cas_config_opid_index: -1
I20260925 21:56:25.733283 8429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true } }
I20260925 21:56:25.733827 8989 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f0b5e1ad65054f35a797c5a6b0e45ed6 (127.2.70.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true } } }
I20260925 21:56:25.734087 8935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:25.734843 8935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 FOLLOWER]: Refusing update from remote peer 1de8b50ece3044268f93c293784dc39b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 21:56:25.734849 8989 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.70.253:43233
I20260925 21:56:25.734926 8989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true } } }
I20260925 21:56:25.734988 8989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:25.735219 8989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:25.735298 8893 master.cc:537] Master@127.2.70.253:43233 shutting down...
W20260925 21:56:25.736963 8414 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 21:56:25.737433 8414 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b -> Peer f0b5e1ad65054f35a797c5a6b0e45ed6 (127.2.70.253:43233): Couldn't send request to peer f0b5e1ad65054f35a797c5a6b0e45ed6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 21:56:25.737820 8893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:25.737910 8893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:25.737955 8893 tablet_replica.cc:330] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: stopping tablet replica
I20260925 21:56:25.750916 8893 master.cc:552] Master@127.2.70.253:43233 shutdown complete.
I20260925 21:56:25.750975 8893 master_runner.cc:305] Clearing existing system tablet
I20260925 21:56:25.751101 8893 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:25.751204 8893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:25.751973 8893 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:25.752516 8993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.752679 8893 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:25.752736 8893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.752758 8893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/wal
uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "aa8a4fdabe05c44e3de791521396ea18"
server_key_iv: "20dab6a18dc1bb577a4f0878456ab67c"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.753537 8893 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:25.755157 8893 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 21:56:25.755230 8893 log.cc:1196] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 21:56:25.755458 8893 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: Deleting consensus metadata
I20260925 21:56:25.755573 8893 master_runner.cc:315] Copying system tablet from 127.2.70.254:43873
I20260925 21:56:25.756163 8893 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: tablet copy: Beginning tablet copy session from remote peer at address 127.2.70.254:43873
I20260925 21:56:25.766144 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.756507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.2.70.254:43873)
0925 21:56:25.756552 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.756554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.760027 (+ 3473us) server_negotiation.cc:407] Connection header received
0925 21:56:25.760093 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.760094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.760106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.760163 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.760565 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.761110 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.762210 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.762410 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.762478 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.763496 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.763498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.763498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.763505 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.764942 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.765343 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.765344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.765345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.765404 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.765618 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.765619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.765620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.765761 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.765824 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.765974 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.766073 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 21:56:25.766189 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.756412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:50542)
0925 21:56:25.759934 (+ 3522us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.759949 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.760028 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.760179 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.760180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.760381 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.760469 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.760472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.761193 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.761194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.762112 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.762115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.762492 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.762493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.762561 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.763014 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.763019 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.763398 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.765036 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.765038 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.765038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.765264 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.765472 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.765473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.765474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.765536 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.765826 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.765826 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.765876 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.765956 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.766074 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3498,"thread_start_us":84,"threads_started":1}
I20260925 21:56:25.766657 8449 tablet_copy_service.cc:140] P 1de8b50ece3044268f93c293784dc39b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f0b5e1ad65054f35a797c5a6b0e45ed6 ({username='oryx', principal='oryx/127.2.70.253@KRBTEST.COM'} at 127.0.0.1:50542)
I20260925 21:56:25.766746 8449 tablet_copy_service.cc:161] P 1de8b50ece3044268f93c293784dc39b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f0b5e1ad65054f35a797c5a6b0e45ed6 at {username='oryx', principal='oryx/127.2.70.253@KRBTEST.COM'} at 127.0.0.1:50542: session id = f0b5e1ad65054f35a797c5a6b0e45ed6-00000000000000000000000000000000
I20260925 21:56:25.767371 8449 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 21:56:25.768006 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 21:56:25.768960 8893 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: tablet copy: Starting download of 0 data blocks...
I20260925 21:56:25.769095 8893 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: tablet copy: Starting download of 1 WAL segments...
I20260925 21:56:25.769918 8893 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 21:56:25.770746 8449 tablet_copy_service.cc:342] P 1de8b50ece3044268f93c293784dc39b: Request end of tablet copy session f0b5e1ad65054f35a797c5a6b0e45ed6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.70.253@KRBTEST.COM'} at 127.0.0.1:50542
I20260925 21:56:25.770798 8449 tablet_copy_service.cc:433] P 1de8b50ece3044268f93c293784dc39b: ending tablet copy session f0b5e1ad65054f35a797c5a6b0e45ed6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f0b5e1ad65054f35a797c5a6b0e45ed6
I20260925 21:56:25.773121 8893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:25.774345 9000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:25.774353 9003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:25.774466 8893 server_base.cc:1034] running on GCE node
W20260925 21:56:25.775970 9001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:25.776145 8893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:25.776298 8893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:25.777436 8893 hybrid_clock.cc:648] HybridClock initialized: now 1790373385777423 us; error 31 us; skew 500 ppm
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:25.778687 8893 init.cc:377] Logged in from keytab as oryx/127.2.70.253@KRBTEST.COM (short username oryx)
I20260925 21:56:25.778926 8893 webserver.cc:460] Webserver started at http://127.2.70.253:46529/ using document root <none> and password file <none>
I20260925 21:56:25.779040 8893 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:25.779066 8893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:25.779659 8893 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:25.780164 9010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.780318 8893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.780364 8893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/wal
uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "aa8a4fdabe05c44e3de791521396ea18"
server_key_iv: "20dab6a18dc1bb577a4f0878456ab67c"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.780421 8893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:25.798969 8893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:25.800534 8893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:25.803284 8893 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.253:43233
I20260925 21:56:25.803299 9062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.253:43233 every 8 connection(s)
I20260925 21:56:25.803593 8893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
I20260925 21:56:25.804283 9063 sys_catalog.cc:263] Verifying existing consensus state
I20260925 21:56:25.804678 9063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: Bootstrap starting.
I20260925 21:56:25.807102 9063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 21:56:25.807286 9063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: Bootstrap complete.
I20260925 21:56:25.807461 9063 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true } }
I20260925 21:56:25.807529 9063 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0b5e1ad65054f35a797c5a6b0e45ed6, State: Initialized, Role: LEARNER
I20260925 21:56:25.807564 9063 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true } }
I20260925 21:56:25.807735 9063 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:25.807731 9067 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true } } }
I20260925 21:56:25.807930 9067 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: This master's current role is: LEARNER
I20260925 21:56:25.809201 9078 catalog_manager.cc:1259] Loaded cluster ID: b18e6ffcbfb144a8a82a34aecb3db065
I20260925 21:56:25.809288 9078 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: loading cluster ID for follower catalog manager: success
I20260925 21:56:25.810386 9078 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: acquiring CA information for follower catalog manager: success
I20260925 21:56:25.810694 9078 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 21:56:25.812935 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.805778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.2.70.253:43233)
0925 21:56:25.805970 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.805972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.805980 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:25.806003 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.806004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.806014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.806080 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.806707 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.807718 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.808781 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.809178 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.809523 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.810627 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.810629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.810629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.810636 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.811968 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.812308 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.812308 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.812309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.812358 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.812531 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.812532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.812532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.812618 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.812681 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.812802 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.812865 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":56,"threads_started":1}
WARNING: no policy specified for oryx/127.2.70.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.252.keytab.
Entry for principal oryx/127.2.70.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.252.keytab.
I20260925 21:56:25.838680 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.252:45607
--webserver_interface=127.2.70.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.252.keytab
--principal=oryx/127.2.70.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:43873,127.2.70.253:43233,127.2.70.252:45607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:25.919015 9081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:25.919241 9081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:25.919272 9081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:25.920646 9081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:25.920715 9081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:25.920737 9081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:25.920754 9081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:25.920769 9081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:25.922436 9081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:43873,127.2.70.253:43233,127.2.70.252:45607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.252.keytab
--principal=oryx/127.2.70.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.252:45607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.70.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:25.922706 9081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:25.922924 9081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:56:25.928004 9081 server_base.cc:1034] running on GCE node
W20260925 21:56:25.928031 9086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:25.927958 9087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:25.927985 9089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:25.928452 9081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:25.928782 9081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:25.929982 9081 hybrid_clock.cc:648] HybridClock initialized: now 1790373385929937 us; error 65 us; skew 500 ppm
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:25.932472 9081 init.cc:377] Logged in from keytab as oryx/127.2.70.252@KRBTEST.COM (short username oryx)
I20260925 21:56:25.932899 9081 webserver.cc:460] Webserver started at http://127.2.70.252:41793/ using document root <none> and password file <none>
I20260925 21:56:25.933089 9081 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:25.933138 9081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:25.933257 9081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:25.934315 9081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data/instance:
uuid: "29d6d14e921f4ef1b54a4dc181f1c577"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "58518d186dd0dd3a4dd6dba81638726e"
server_key_iv: "95437e5ea0f31b51f09e650b3dd247ef"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.934607 9081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/wal/instance:
uuid: "29d6d14e921f4ef1b54a4dc181f1c577"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "58518d186dd0dd3a4dd6dba81638726e"
server_key_iv: "95437e5ea0f31b51f09e650b3dd247ef"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.935822 9081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:25.936631 9096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:25.936781 9081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:25.936870 9081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/wal
uuid: "29d6d14e921f4ef1b54a4dc181f1c577"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "58518d186dd0dd3a4dd6dba81638726e"
server_key_iv: "95437e5ea0f31b51f09e650b3dd247ef"
server_key_version: "encryptionkey@0"
I20260925 21:56:25.936939 9081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:25.950217 9081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:25.952378 9081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:25.952517 9081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:25.956377 9081 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.252:45607
I20260925 21:56:25.956465 9148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.252:45607 every 8 connection(s)
I20260925 21:56:25.956756 9081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
I20260925 21:56:25.957705 9149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:25.959241 9149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } has no permanent_uuid. Determining permanent_uuid...
I20260925 21:56:25.963688 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 9081
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.252@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
I20260925 21:56:25.968394 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.960011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.2.70.254:43873)
0925 21:56:25.960054 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.960056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.960203 (+ 147us) server_negotiation.cc:407] Connection header received
0925 21:56:25.960658 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.960659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.960668 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.960723 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.961278 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.961793 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.962788 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.962964 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.963007 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.965397 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.965399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.965399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.965406 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.966878 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.967380 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.967381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.967381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.967440 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.967703 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.967704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.967705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.967854 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.967919 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.968221 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.968299 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 21:56:25.968458 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.959933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:50558)
0925 21:56:25.960088 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.960130 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.960583 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.960778 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.960781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.961029 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.961218 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.961221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.961869 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.961870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.962725 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.962727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.963037 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.963037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.963142 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.963628 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.963638 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.965309 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.966981 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.966989 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.966992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.967288 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.967508 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.967509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.967509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.967642 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.967928 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.967934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.968089 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.968175 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.968292 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260925 21:56:25.969255 9149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.252@KRBTEST.COM
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.252@KRBTEST.COM for oryx/127.2.70.253@KRBTEST.COM
I20260925 21:56:25.972765 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.964338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.2.70.252:45607)
0925 21:56:25.964464 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.964467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.964480 (+ 13us) server_negotiation.cc:407] Connection header received
0925 21:56:25.964498 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.964499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.964514 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.964575 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.965269 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.965990 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.967026 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.967462 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.967524 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.970151 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.970159 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.970159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.970165 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.971533 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.972047 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.972050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.972050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.972102 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.972261 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.972261 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.972262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.972390 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.972515 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.972598 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.972665 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260925 21:56:25.975252 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.969514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43233 (local address 127.0.0.1:52536)
0925 21:56:25.969559 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.969570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.969610 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.969736 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.969737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.969864 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.969918 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.969920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.970554 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.970555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.971083 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.971084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.971410 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.971411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.971478 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.971771 (+ 293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.971775 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.973003 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.974468 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.974469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.974469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.974633 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.974773 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.974774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.974774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.974814 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.975054 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.975055 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.975086 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.975129 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.975192 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 21:56:25.975267 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.969597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.2.70.253:43233)
0925 21:56:25.969637 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.969639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.969647 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:25.969668 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.969669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.969677 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.969726 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.969991 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.970447 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.971158 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.971355 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.971408 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.973084 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.973085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.973086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.973092 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.974409 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.974688 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.974689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.974689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.974733 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.974847 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.974847 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.974847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.974934 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.975060 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.975147 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.975209 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 21:56:25.975589 9149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 21:56:25 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373385, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.252@KRBTEST.COM for oryx/127.2.70.252@KRBTEST.COM
I20260925 21:56:25.981637 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.975885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.2.70.252:45607)
0925 21:56:25.975913 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.975914 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.975922 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:25.975981 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.975982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.975990 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.976032 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.976218 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.976653 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.977269 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.977458 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.977514 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.979037 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.979039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.979039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.979045 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.980493 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.980834 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.980835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.980835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.980888 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.981148 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.981149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.981150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.981279 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.981332 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.981442 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.981523 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12}
I20260925 21:56:25.981681 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.975818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:45607 (local address 127.0.0.1:51940)
0925 21:56:25.975869 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.975880 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.975910 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.976037 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.976038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.976145 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.976173 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.976174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.976718 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.976719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.977188 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.977189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.977518 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.977519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.977577 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.977864 (+ 287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.977868 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.978941 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.980574 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.980575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.980575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.980769 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.981016 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.981017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.981017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.981074 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.981334 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.981334 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.981367 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.981417 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.981569 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":54}
I20260925 21:56:25.983500 9149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: Bootstrap starting.
I20260925 21:56:25.984381 9149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:25.984637 9149 log.cc:824] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:25.985339 9149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: No bootstrap required, opened a new log
I20260925 21:56:25.986733 9149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } }
I20260925 21:56:25.986866 9149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:25.986898 9149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d6d14e921f4ef1b54a4dc181f1c577, State: Initialized, Role: FOLLOWER
I20260925 21:56:25.987022 9149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } }
I20260925 21:56:25.987264 9154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } } }
I20260925 21:56:25.987313 9149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:25.987357 9154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 21:56:25.989012 9165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:56:25.989075 9165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:56:25.991190 9081 master_runner.cc:428] Detected that this master 29d6d14e921f4ef1b54a4dc181f1c577 is joining an existing cluster
I20260925 21:56:25.991231 9081 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.70.252:45607
I20260925 21:56:25.998262 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.992502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.2.70.253:43233)
0925 21:56:25.992541 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.992542 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.992549 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:56:25.992568 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.992569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.992577 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.992630 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.992917 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.993670 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.994564 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.994777 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.994833 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.995941 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:25.995942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:25.995942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:25.995948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:25.997176 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.997530 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.997531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.997531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.997577 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:25.997726 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:25.997726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:25.997726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:25.997831 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:25.997892 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:25.998093 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 21:56:25.998170 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 21:56:25.998607 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.992165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43233 (local address 127.0.0.1:52538)
0925 21:56:25.992463 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.992479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.992516 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.992630 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.992631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.992787 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.992847 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.992849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.993755 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.993756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.994495 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.994497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.994838 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.994839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.994897 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:25.995441 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:25.995446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:25.995870 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:25.997251 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.997252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.997253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.997465 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.997615 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:25.997615 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:25.997616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:25.997676 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:25.997922 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:25.997923 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:25.997969 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 21:56:25.998069 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 21:56:25.998206 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":78,"threads_started":1}
I20260925 21:56:26.003778 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.992360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:45607 (local address 127.0.0.1:51944)
0925 21:56:25.998698 (+ 6338us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.998710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.998792 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.998904 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.998904 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.999056 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.999102 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.999104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.999682 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.999682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.000205 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.000207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.000492 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.000493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.000543 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.001018 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.001022 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.001280 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.002792 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.002793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.002793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.002942 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.003163 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.003164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.003164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.003216 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.003440 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.003441 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.003477 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.003557 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.003649 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6325,"thread_start_us":65,"threads_started":1}
I20260925 21:56:26.003778 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.992503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.2.70.252:45607)
0925 21:56:25.992533 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.992534 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.998770 (+ 6236us) server_negotiation.cc:407] Connection header received
0925 21:56:25.998843 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.998844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.998852 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.998893 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.999161 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.999618 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.000267 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.000441 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.000498 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.001374 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.001375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.001375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.001382 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.002715 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.003064 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.003065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.003065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.003110 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.003263 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.003264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.003265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.003389 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.003442 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.003583 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.003695 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":16}
I20260925 21:56:26.019726 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.992149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:50566)
0925 21:56:25.992287 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:25.992297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:25.992341 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:25.992674 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:25.992675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:25.992884 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:25.992951 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.992953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.993825 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.993826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.994687 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:25.994689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.995078 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.995079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:25.995168 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.016440 (+ 21272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.016446 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.016898 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.018784 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.018786 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.018786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.018961 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.019149 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.019149 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.019149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.019197 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.019442 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.019443 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.019486 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.019554 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.019645 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260925 21:56:26.019764 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:25.992188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.2.70.254:43873)
0925 21:56:25.992347 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:25.992349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:25.992361 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:56:25.992385 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:25.992386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:25.992397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:25.992451 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:25.993038 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.993754 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.994766 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:25.995016 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:25.995070 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.017023 (+ 21953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.017026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.017026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.017035 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.018681 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.019029 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.019030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.019031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.019090 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.019246 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.019246 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.019247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.019386 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.019461 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.019590 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.019671 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260925 21:56:26.024304 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 8491
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373384, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.254@KRBTEST.COM for oryx/127.2.70.252@KRBTEST.COM
I20260925 21:56:26.029239 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.021699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.2.70.252:45607)
0925 21:56:26.021739 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.021741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.021748 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:56:26.021810 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.021810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.021818 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.021858 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.022137 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.022586 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.023607 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.023855 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.023926 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.025927 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.025929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.025929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.025936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.027582 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.028466 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.028467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.028468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.028526 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.028728 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.028729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.028729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.028855 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.028917 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.029067 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.029146 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 21:56:26.029245 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.021640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:45607 (local address 127.0.0.1:51956)
0925 21:56:26.021689 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.021700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.021743 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.021863 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.021864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.022019 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:26.022071 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.022074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.022650 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.022650 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.023483 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.023485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.023926 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.023927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.024000 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.024390 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.024396 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.025842 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.027684 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.027685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.027685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.027905 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.028597 (+ 692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.028598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.028598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.028666 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.028914 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.028915 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.028969 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.029036 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.029135 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 21:56:26.029940 8429 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.70.252:45607: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577"
member_type: NON_VOTER
last_known_addr {
host: "127.2.70.252"
port: 45607
}
attrs {
promote: true
}
}
dest_uuid: "1de8b50ece3044268f93c293784dc39b"
cas_config_opid_index: 5
I20260925 21:56:26.030185 8429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } }
I20260925 21:56:26.030316 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:43971
--local_ip_for_outbound_sockets=127.2.70.193
--tserver_master_addrs=127.2.70.254:43873
--webserver_port=42725
--webserver_interface=127.2.70.193
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab
--principal=oryx/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.70.254:43873,127.2.70.253:43233
--tserver_master_addrs=127.2.70.254:43873,127.2.70.253:43233,127.2.70.252:45607 with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:56:26.030675 8989 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 29d6d14e921f4ef1b54a4dc181f1c577 (127.2.70.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.030920 9123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:26.031626 8989 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.70.252:45607
I20260925 21:56:26.031701 8989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.031760 8989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:26.031814 9123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 FOLLOWER]: Refusing update from remote peer 1de8b50ece3044268f93c293784dc39b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 21:56:26.032064 9081 master.cc:537] Master@127.2.70.252:45607 shutting down...
I20260925 21:56:26.032085 8988 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 21:56:26.032697 8413 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b -> Peer 29d6d14e921f4ef1b54a4dc181f1c577 (127.2.70.252:45607): Couldn't send request to peer 29d6d14e921f4ef1b54a4dc181f1c577. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 21:56:26.035144 9081 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:26.035236 9081 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:26.035277 9081 tablet_replica.cc:330] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: stopping tablet replica
I20260925 21:56:26.037539 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.031729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.2.70.253:43233)
0925 21:56:26.031774 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.031776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.033865 (+ 2089us) server_negotiation.cc:407] Connection header received
0925 21:56:26.033959 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.033960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.033986 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.034025 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:26.034216 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.034929 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.036390 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.036991 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.037042 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.037124 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.037406 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.037453 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 21:56:26.037917 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.030611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43233 (local address 127.0.0.1:52554)
0925 21:56:26.033803 (+ 3192us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.033816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.033894 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.034101 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.034102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.034103 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:26.034147 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.034149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.035028 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.035029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.036275 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.036278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.037207 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.037208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.037291 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.037320 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.037803 (+ 483us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.037855 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3173}
I20260925 21:56:26.038358 9037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 LEARNER]: Refusing update from remote peer 1de8b50ece3044268f93c293784dc39b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 21:56:26.038580 8988 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 21:56:26.039156 9037 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 29d6d14e921f4ef1b54a4dc181f1c577 (127.2.70.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.040509 9067 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1de8b50ece3044268f93c293784dc39b. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.040599 9067 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: This master's current role is: LEARNER
I20260925 21:56:26.040634 9067 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: NON_VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: true } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.040653 8988 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b: attempting to promote NON_VOTER f0b5e1ad65054f35a797c5a6b0e45ed6 to VOTER
I20260925 21:56:26.040673 9067 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: This master's current role is: LEARNER
I20260925 21:56:26.040868 8988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } }
I20260925 21:56:26.041460 9037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 LEARNER]: Refusing update from remote peer 1de8b50ece3044268f93c293784dc39b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 21:56:26.041672 9177 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 21:56:26.041719 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.041610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:45607 (local address 127.0.0.1:51962)
0925 21:56:26.041645 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.041685 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:45607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 21:56:26.041994 8413 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b -> Peer 29d6d14e921f4ef1b54a4dc181f1c577 (127.2.70.252:45607): Couldn't send request to peer 29d6d14e921f4ef1b54a4dc181f1c577. Status: Network error: Client connection negotiation failed: client connection to 127.2.70.252:45607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 21:56:26.042481 8989 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f0b5e1ad65054f35a797c5a6b0e45ed6 (127.2.70.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.042600 9037 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f0b5e1ad65054f35a797c5a6b0e45ed6 (127.2.70.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.043515 8988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.043591 8988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:26.043527 8989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.043769 8989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:26.044090 9176 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.044190 9176 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 21:56:26.051384 9081 master.cc:552] Master@127.2.70.252:45607 shutdown complete.
I20260925 21:56:26.051440 9081 master_runner.cc:305] Clearing existing system tablet
I20260925 21:56:26.051601 9081 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:26.051647 9081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:26.052402 9081 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.052934 9180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.053040 9081 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:26.053086 9081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.053107 9081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/wal
uuid: "29d6d14e921f4ef1b54a4dc181f1c577"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "58518d186dd0dd3a4dd6dba81638726e"
server_key_iv: "95437e5ea0f31b51f09e650b3dd247ef"
server_key_version: "encryptionkey@0"
I20260925 21:56:26.053861 9081 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:26.055629 9081 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 21:56:26.055701 9081 log.cc:1196] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 21:56:26.055945 9081 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: Deleting consensus metadata
I20260925 21:56:26.056083 9081 master_runner.cc:315] Copying system tablet from 127.2.70.254:43873
I20260925 21:56:26.056710 9081 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: tablet copy: Beginning tablet copy session from remote peer at address 127.2.70.254:43873
I20260925 21:56:26.063684 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.057073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.2.70.254:43873)
0925 21:56:26.057117 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.057119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.057193 (+ 74us) server_negotiation.cc:407] Connection header received
0925 21:56:26.057288 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.057289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.057299 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.057373 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.057735 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.058453 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.059452 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.059732 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.059806 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.060911 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.060913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.060913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.060921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.062522 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.062900 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.062902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.062902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.062965 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.063151 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.063152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.063152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.063291 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.063360 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.063506 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.063586 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 21:56:26.063689 9186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.057001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:50570)
0925 21:56:26.057124 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.057134 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.057232 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.057386 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.057386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.057593 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:26.057651 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.057654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.058539 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.058540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.059352 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.059355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.059811 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.059812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.059915 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.060418 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.060423 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.060814 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.062610 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.062611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.062612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.062826 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.063029 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.063030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.063030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.063092 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.063353 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.063354 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.063403 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.063473 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.063585 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260925 21:56:26.064061 8449 tablet_copy_service.cc:140] P 1de8b50ece3044268f93c293784dc39b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 29d6d14e921f4ef1b54a4dc181f1c577 ({username='oryx', principal='oryx/127.2.70.252@KRBTEST.COM'} at 127.0.0.1:50570)
I20260925 21:56:26.064119 8449 tablet_copy_service.cc:161] P 1de8b50ece3044268f93c293784dc39b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 29d6d14e921f4ef1b54a4dc181f1c577 at {username='oryx', principal='oryx/127.2.70.252@KRBTEST.COM'} at 127.0.0.1:50570: session id = 29d6d14e921f4ef1b54a4dc181f1c577-00000000000000000000000000000000
I20260925 21:56:26.064711 8449 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 21:56:26.065371 9081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 21:56:26.066717 9081 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: tablet copy: Starting download of 0 data blocks...
I20260925 21:56:26.066870 9081 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: tablet copy: Starting download of 1 WAL segments...
I20260925 21:56:26.067693 9081 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 21:56:26.068751 8449 tablet_copy_service.cc:342] P 1de8b50ece3044268f93c293784dc39b: Request end of tablet copy session 29d6d14e921f4ef1b54a4dc181f1c577-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.70.252@KRBTEST.COM'} at 127.0.0.1:50570
I20260925 21:56:26.068819 8449 tablet_copy_service.cc:433] P 1de8b50ece3044268f93c293784dc39b: ending tablet copy session 29d6d14e921f4ef1b54a4dc181f1c577-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 29d6d14e921f4ef1b54a4dc181f1c577
I20260925 21:56:26.071384 9081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:26.072726 9190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:26.072984 9081 server_base.cc:1034] running on GCE node
W20260925 21:56:26.073042 9187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:26.072983 9188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:26.073316 9081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:26.073491 9081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:26.074594 9081 hybrid_clock.cc:648] HybridClock initialized: now 1790373386074585 us; error 26 us; skew 500 ppm
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:26.076077 9081 init.cc:377] Logged in from keytab as oryx/127.2.70.252@KRBTEST.COM (short username oryx)
I20260925 21:56:26.076335 9081 webserver.cc:460] Webserver started at http://127.2.70.252:36141/ using document root <none> and password file <none>
I20260925 21:56:26.076463 9081 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:26.076502 9081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:26.077219 9081 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.077834 9197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.078011 9081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.078065 9081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/wal
uuid: "29d6d14e921f4ef1b54a4dc181f1c577"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "58518d186dd0dd3a4dd6dba81638726e"
server_key_iv: "95437e5ea0f31b51f09e650b3dd247ef"
server_key_version: "encryptionkey@0"
I20260925 21:56:26.078130 9081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:26.088482 9081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:26.090142 9081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:26.093154 9081 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.252:45607
I20260925 21:56:26.093556 9081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
I20260925 21:56:26.093895 9249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.252:45607 every 8 connection(s)
I20260925 21:56:26.096664 9250 sys_catalog.cc:263] Verifying existing consensus state
I20260925 21:56:26.097199 9250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: Bootstrap starting.
I20260925 21:56:26.100164 9250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 21:56:26.100365 9250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: Bootstrap complete.
I20260925 21:56:26.100555 9250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } }
I20260925 21:56:26.100648 9250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29d6d14e921f4ef1b54a4dc181f1c577, State: Initialized, Role: LEARNER
I20260925 21:56:26.100682 9250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } }
I20260925 21:56:26.100871 9250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:26.100878 9253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.101094 9253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: This master's current role is: LEARNER
I20260925 21:56:26.102272 9264 catalog_manager.cc:1259] Loaded cluster ID: b18e6ffcbfb144a8a82a34aecb3db065
I20260925 21:56:26.102357 9264 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: loading cluster ID for follower catalog manager: success
I20260925 21:56:26.103598 9264 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: acquiring CA information for follower catalog manager: success
I20260925 21:56:26.103941 9264 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 21:56:26.126384 9174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:26.126585 9174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:26.126607 9174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:26.127948 9174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:26.128000 9174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:26.128027 9174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:26.129525 9174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab
--principal=oryx/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:43971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=42725
--tserver_master_addrs=127.2.70.254:43873,127.2.70.253:43233,127.2.70.252:45607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:26.129747 9174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:26.129969 9174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:26.134981 9272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:26.134927 9269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:26.135319 9270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:26.135428 9174 server_base.cc:1034] running on GCE node
I20260925 21:56:26.135632 9174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:26.135888 9174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:26.137077 9174 hybrid_clock.cc:648] HybridClock initialized: now 1790373386137059 us; error 33 us; skew 500 ppm
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:26.139715 9174 init.cc:377] Logged in from keytab as oryx/127.2.70.193@KRBTEST.COM (short username oryx)
I20260925 21:56:26.140210 9174 webserver.cc:460] Webserver started at http://127.2.70.193:42725/ using document root <none> and password file <none>
I20260925 21:56:26.140432 9174 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:26.140501 9174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:26.141883 9174 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:26.142871 9279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.143038 9174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:26.143133 9174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "0882610850c9425ea77ae50c3a43dfbb"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "ddf7d33858db8eb4ecd9fdc3ee04ea22"
server_key_iv: "f25dbbedeb5703c79dc4107f6c95512c"
server_key_version: "encryptionkey@0"
I20260925 21:56:26.143498 9174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:26.153470 9174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:26.155663 9174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:26.155793 9174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:26.156046 9174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:26.156364 9174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:26.156405 9174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.156435 9174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:26.156451 9174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.162182 9174 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:43971
I20260925 21:56:26.162230 9392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:43971 every 8 connection(s)
I20260925 21:56:26.162559 9174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:26.166471 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 9174
I20260925 21:56:26.166620 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 8625
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.252@KRBTEST.COM
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.253@KRBTEST.COM
I20260925 21:56:26.173671 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:35501
--local_ip_for_outbound_sockets=127.2.70.194
--tserver_master_addrs=127.2.70.254:43873
--webserver_port=34419
--webserver_interface=127.2.70.194
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab
--principal=oryx/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.70.254:43873,127.2.70.253:43233
--tserver_master_addrs=127.2.70.254:43873,127.2.70.253:43233,127.2.70.252:45607 with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:56:26.174818 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.163263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:37867 (local address 127.2.70.253:43233)
0925 21:56:26.163307 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.163310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.163507 (+ 197us) server_negotiation.cc:407] Connection header received
0925 21:56:26.164235 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.164237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.164249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.164279 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.165197 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.165964 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.168526 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.168820 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.168884 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.171531 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.171533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.171533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.171541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.173199 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.173681 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.173682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.173683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.173741 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.174115 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.174116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.174116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.174259 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.174295 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.174646 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.174729 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 21:56:26.175032 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.163246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:45607 (local address 127.2.70.193:41227)
0925 21:56:26.163719 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.163728 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.164123 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.164645 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.164647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.164873 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:26.165087 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.165091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.166241 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.166242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.167287 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.167289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.167813 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.167814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.167971 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.168774 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.168784 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.171474 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.173478 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.173479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.173479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.173703 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.173899 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.173900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.173900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.174046 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.174337 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.174342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.174511 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.174610 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.174786 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":27,"thread_start_us":88,"threads_started":1}
I20260925 21:56:26.175163 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.163608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:58933 (local address 127.2.70.254:43873)
0925 21:56:26.163655 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.163657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.163892 (+ 235us) server_negotiation.cc:407] Connection header received
0925 21:56:26.164236 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.164237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.164248 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.164279 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.166051 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.166719 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.167631 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.167990 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.168030 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.171580 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.171581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.171581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.171588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.173702 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.174195 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.174196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.174196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.174244 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.174436 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.174437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.174438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.174659 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.174725 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.175010 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.175084 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 21:56:26.175288 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.163245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43233 (local address 127.2.70.193:37867)
0925 21:56:26.163396 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.163437 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.164120 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.164370 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.164375 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.164878 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:26.165089 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.165094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.166118 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.166118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.168427 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.168429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.168937 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.168938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.169011 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.169339 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.169344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.171415 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.173271 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.173278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.173280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.173600 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.173933 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.173934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.173934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.174046 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.174334 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.174339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.174502 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.175074 (+ 572us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.175186 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":52,"thread_start_us":86,"threads_started":1}
I20260925 21:56:26.175458 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.163237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:41227 (local address 127.2.70.252:45607)
0925 21:56:26.163344 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.163346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.163787 (+ 441us) server_negotiation.cc:407] Connection header received
0925 21:56:26.164207 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.164209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.164224 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.164288 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.165179 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.166171 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.167376 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.167691 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.167749 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.171573 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.171575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.171575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.171583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.173391 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.173779 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.173780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.173780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.173840 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.174124 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.174126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.174126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.174260 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.174315 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.175263 (+ 948us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.175358 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":49,"threads_started":1}
I20260925 21:56:26.175649 9395 heartbeater.cc:347] Connected to a master server at 127.2.70.253:43233
I20260925 21:56:26.175731 9393 heartbeater.cc:347] Connected to a master server at 127.2.70.252:45607
I20260925 21:56:26.175791 9393 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:26.175989 9393 heartbeater.cc:511] Master 127.2.70.252:45607 requested a full tablet report, sending...
I20260925 21:56:26.175736 9395 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:26.176234 9395 heartbeater.cc:511] Master 127.2.70.253:43233 requested a full tablet report, sending...
I20260925 21:56:26.176294 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.163534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.193:58933)
0925 21:56:26.163813 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.163823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.164131 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.164381 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.164382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.164875 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:26.165089 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.165094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.166806 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.166807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.167538 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.167541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.168103 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.168104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.168162 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.168837 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.168841 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.171474 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.173897 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.173898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.173898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.174119 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.174309 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.174309 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.174310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.174359 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.174771 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.174771 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.174804 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.175875 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.175976 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":5,"thread_start_us":53,"threads_started":1}
I20260925 21:56:26.176580 9394 heartbeater.cc:347] Connected to a master server at 127.2.70.254:43873
I20260925 21:56:26.176615 9394 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:26.176594 9027 ts_manager.cc:194] Registered new tserver with Master: 0882610850c9425ea77ae50c3a43dfbb (127.2.70.193:43971)
I20260925 21:56:26.176609 9205 ts_manager.cc:194] Registered new tserver with Master: 0882610850c9425ea77ae50c3a43dfbb (127.2.70.193:43971)
I20260925 21:56:26.176671 9394 heartbeater.cc:511] Master 127.2.70.254:43873 requested a full tablet report, sending...
I20260925 21:56:26.176959 8429 ts_manager.cc:194] Re-registered known tserver with Master: 0882610850c9425ea77ae50c3a43dfbb (127.2.70.193:43971)
I20260925 21:56:26.177448 8429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:58933
W20260925 21:56:26.255438 9401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:26.255645 9401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:26.255667 9401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:26.256839 9401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:26.256884 9401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:26.256911 9401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:26.258363 9401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab
--principal=oryx/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:35501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=34419
--tserver_master_addrs=127.2.70.254:43873,127.2.70.253:43233,127.2.70.252:45607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:26.258553 9401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:26.258764 9401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:26.263688 9407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:26.263671 9409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:26.263782 9401 server_base.cc:1034] running on GCE node
W20260925 21:56:26.263671 9406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:26.264106 9401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:26.264333 9401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:26.265487 9401 hybrid_clock.cc:648] HybridClock initialized: now 1790373386265470 us; error 30 us; skew 500 ppm
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:26.267994 9401 init.cc:377] Logged in from keytab as oryx/127.2.70.194@KRBTEST.COM (short username oryx)
I20260925 21:56:26.268428 9401 webserver.cc:460] Webserver started at http://127.2.70.194:34419/ using document root <none> and password file <none>
I20260925 21:56:26.268664 9401 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:26.268729 9401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:26.270072 9401 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:26.270705 9416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.270864 9401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:26.270951 9401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "0be7ac9f2c9d4849b2f19f16a277448d"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "3c3628114ffd0133460cc8c9ec1455a3"
server_key_iv: "8b8a335468d20924d87e42f701c415c3"
server_key_version: "encryptionkey@0"
I20260925 21:56:26.271303 9401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:26.296934 9401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:26.299067 9401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:26.299199 9401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:26.299405 9401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:26.299695 9401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:26.299731 9401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.299760 9401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:26.299784 9401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.305244 9401 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:35501
I20260925 21:56:26.305310 9529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:35501 every 8 connection(s)
I20260925 21:56:26.305662 9401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:26.310019 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 9401
I20260925 21:56:26.310184 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 8759
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.252@KRBTEST.COM
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.253@KRBTEST.COM
I20260925 21:56:26.317337 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:39491
--local_ip_for_outbound_sockets=127.2.70.195
--tserver_master_addrs=127.2.70.254:43873
--webserver_port=34579
--webserver_interface=127.2.70.195
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab
--principal=oryx/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.70.254:43873,127.2.70.253:43233
--tserver_master_addrs=127.2.70.254:43873,127.2.70.253:43233,127.2.70.252:45607 with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:56:26.318240 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.307737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:45291 (local address 127.2.70.252:45607)
0925 21:56:26.307778 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.307781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.307911 (+ 130us) server_negotiation.cc:407] Connection header received
0925 21:56:26.308068 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.308070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.308086 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.308188 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.308991 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.309607 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.311158 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.311467 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.311505 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.315234 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.315235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.315235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.315244 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.316693 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.317173 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.317175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.317175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.317232 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.317498 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.317499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.317500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.317629 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.317669 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.318062 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.318153 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 21:56:26.318238 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.306560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:52891 (local address 127.2.70.253:43233)
0925 21:56:26.306612 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.306614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.307437 (+ 823us) server_negotiation.cc:407] Connection header received
0925 21:56:26.307901 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.307903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.307916 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.307947 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.309021 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.309773 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.311379 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.311641 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.311678 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.315345 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.315346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.315347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.315354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.316967 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.317442 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.317443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.317443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.317503 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.317676 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.317676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.317677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.317802 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.317841 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.318060 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.318145 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 21:56:26.318403 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.306506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:52483 (local address 127.2.70.254:43873)
0925 21:56:26.306580 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.306582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.306953 (+ 371us) server_negotiation.cc:407] Connection header received
0925 21:56:26.309074 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.309076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.309089 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.309118 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.309480 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.310137 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.311146 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.311429 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.311478 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.314710 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.314711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.314712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.314720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.316952 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.317675 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.317676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.317676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.317728 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.317910 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.317911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.317911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.318034 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.318072 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.318248 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.318324 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 21:56:26.318835 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.307648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:45607 (local address 127.2.70.194:45291)
0925 21:56:26.307861 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.307870 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.307931 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.308192 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.308193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.308597 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:26.308886 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.308892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.309716 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.309716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.311055 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.311065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.311578 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.311578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.311704 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.312338 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.312349 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.315136 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.316776 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.316784 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.316786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.317088 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.317296 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.317297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.317297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.317423 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.317721 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.317727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.317909 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.318327 (+ 418us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.318592 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":40,"thread_start_us":68,"threads_started":1}
I20260925 21:56:26.319037 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.306443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.194:52483)
0925 21:56:26.306785 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.306856 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.307716 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.309175 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.309176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.309327 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:26.309396 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.309398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.310223 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.310224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.311054 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.311056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.311557 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.311558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.311704 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.312372 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.312377 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.314616 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.317050 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.317051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.317051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.317361 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.317794 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.317795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.317795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.317844 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.318112 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.318112 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.318146 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.318690 (+ 544us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.318791 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":82,"threads_started":1}
I20260925 21:56:26.319206 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.306647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43233 (local address 127.2.70.194:52891)
0925 21:56:26.307368 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.307379 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.307796 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.308137 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.308141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.308739 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:26.308886 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.308892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.309850 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.309852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.311299 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.311301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.311744 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.311745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.311802 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.312464 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.312468 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.315239 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.317050 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.317051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.317051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.317361 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.317566 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.317566 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.317567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.317616 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.317882 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.317883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.317949 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.318353 (+ 404us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.318845 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":385,"thread_start_us":72,"threads_started":1}
I20260925 21:56:26.319265 9530 heartbeater.cc:347] Connected to a master server at 127.2.70.252:45607
I20260925 21:56:26.319353 9530 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:26.319468 9531 heartbeater.cc:347] Connected to a master server at 127.2.70.254:43873
I20260925 21:56:26.319512 9531 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:26.319532 9532 heartbeater.cc:347] Connected to a master server at 127.2.70.253:43233
I20260925 21:56:26.319553 9530 heartbeater.cc:511] Master 127.2.70.252:45607 requested a full tablet report, sending...
I20260925 21:56:26.319574 9531 heartbeater.cc:511] Master 127.2.70.254:43873 requested a full tablet report, sending...
I20260925 21:56:26.319576 9532 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:26.319649 9532 heartbeater.cc:511] Master 127.2.70.253:43233 requested a full tablet report, sending...
I20260925 21:56:26.319993 8429 ts_manager.cc:194] Re-registered known tserver with Master: 0be7ac9f2c9d4849b2f19f16a277448d (127.2.70.194:35501)
I20260925 21:56:26.320029 9205 ts_manager.cc:194] Registered new tserver with Master: 0be7ac9f2c9d4849b2f19f16a277448d (127.2.70.194:35501)
I20260925 21:56:26.320201 9027 ts_manager.cc:194] Registered new tserver with Master: 0be7ac9f2c9d4849b2f19f16a277448d (127.2.70.194:35501)
I20260925 21:56:26.320462 8429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:52483
W20260925 21:56:26.399447 9537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:26.399655 9537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:26.399678 9537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:26.400892 9537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:26.400964 9537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:26.401002 9537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:26.402566 9537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab
--principal=oryx/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:39491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=34579
--tserver_master_addrs=127.2.70.254:43873,127.2.70.253:43233,127.2.70.252:45607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:26.402768 9537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:26.402946 9537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:26.408134 9543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:26.408145 9542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:26.408174 9545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:26.408330 9537 server_base.cc:1034] running on GCE node
I20260925 21:56:26.408519 9537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:26.408746 9537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:26.409896 9537 hybrid_clock.cc:648] HybridClock initialized: now 1790373386409884 us; error 27 us; skew 500 ppm
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:26.412534 9537 init.cc:377] Logged in from keytab as oryx/127.2.70.195@KRBTEST.COM (short username oryx)
I20260925 21:56:26.412981 9537 webserver.cc:460] Webserver started at http://127.2.70.195:34579/ using document root <none> and password file <none>
I20260925 21:56:26.413199 9537 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:26.413280 9537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:26.414618 9537 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:26.415266 9552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.415432 9537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:26.415530 9537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "d15c2de96cec44be9ed402dbaead6aba"
format_stamp: "Formatted at 2026-09-25 21:56:25 on dist-test-slave-2g0f"
server_key: "6fb36d870c9f4fa49dc9414b8f4b86a1"
server_key_iv: "c403804d50555235f2c65d08e6a62f5f"
server_key_version: "encryptionkey@0"
I20260925 21:56:26.415916 9537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:26.431136 9537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:26.433231 9537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:26.433369 9537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:26.433629 9537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:26.433928 9537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:26.433961 9537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.433992 9537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:26.434010 9537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:26.439711 9537 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:39491
I20260925 21:56:26.439760 9665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:39491 every 8 connection(s)
I20260925 21:56:26.440115 9537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:26.443710 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 9537
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.252@KRBTEST.COM
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.253@KRBTEST.COM
Sep 25 21:56:26 dist-test-slave-2g0f krb5kdc[8384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373386, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
I20260925 21:56:26.452386 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.440833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:34161 (local address 127.2.70.253:43233)
0925 21:56:26.440878 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.440880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.441208 (+ 328us) server_negotiation.cc:407] Connection header received
0925 21:56:26.441783 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.441784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.441796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.441826 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.442622 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.443373 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.444846 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.445135 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.445191 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.448450 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.448451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.448451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.448458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.450137 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.451177 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.451178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.451178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.451232 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.451535 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.451536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.451536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.451669 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.451725 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.452212 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.452285 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 21:56:26.452857 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.440844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43233 (local address 127.2.70.195:34161)
0925 21:56:26.441028 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.441074 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.441667 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.442258 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.442259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.442394 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:26.442509 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.442514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.443487 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.443488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.444760 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.444762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.445251 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.445252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.445383 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.445954 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.445964 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.448350 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.450527 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.450534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.450536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.450846 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.451305 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.451306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.451307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.451453 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.451749 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.451755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.451938 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.452045 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.452593 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":5,"spinlock_wait_cycles":384,"thread_start_us":89,"threads_started":1}
I20260925 21:56:26.453357 9668 heartbeater.cc:347] Connected to a master server at 127.2.70.253:43233
I20260925 21:56:26.453393 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.441254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:49609 (local address 127.2.70.252:45607)
0925 21:56:26.441298 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.441300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.441425 (+ 125us) server_negotiation.cc:407] Connection header received
0925 21:56:26.441756 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.441758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.441769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.441829 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.442596 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.443373 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.445143 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.445446 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.445517 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.449121 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.449123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.449123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.449130 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.451527 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.452619 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.452620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.452620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.452674 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.452856 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.452857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.452857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.453001 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.453055 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.453233 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.453310 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 21:56:26.453454 9668 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:26.453648 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.441133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:45607 (local address 127.2.70.195:49609)
0925 21:56:26.441354 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.441363 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.441674 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.441913 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.441914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.442303 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:26.442508 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.442514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.443506 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.443507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.445053 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.445055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.445603 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.445604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.445667 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.446071 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.446076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.449024 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.451604 (+ 2580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.451605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.451605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.452536 (+ 931us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.452740 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.452741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.452741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.452794 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.453086 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.453087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.453127 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.453463 (+ 336us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.453563 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":80,"thread_start_us":61,"threads_started":1}
I20260925 21:56:26.453655 9668 heartbeater.cc:511] Master 127.2.70.253:43233 requested a full tablet report, sending...
I20260925 21:56:26.454113 9027 ts_manager.cc:194] Registered new tserver with Master: d15c2de96cec44be9ed402dbaead6aba (127.2.70.195:39491)
I20260925 21:56:26.454253 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.445813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.2.70.252:45607)
0925 21:56:26.446220 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.446222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.447367 (+ 1145us) server_negotiation.cc:407] Connection header received
0925 21:56:26.447386 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.447387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.447397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.447446 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.448136 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.448817 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.449732 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.450009 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.450062 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.451427 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.451428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.451429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.451435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.452867 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.453234 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.453235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.453235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.453287 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.453532 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.453533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.453534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.453647 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.453733 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.454092 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.454167 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":55,"threads_started":1}
I20260925 21:56:26.454298 9666 heartbeater.cc:347] Connected to a master server at 127.2.70.252:45607
I20260925 21:56:26.454339 9666 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:26.454396 9666 heartbeater.cc:511] Master 127.2.70.252:45607 requested a full tablet report, sending...
I20260925 21:56:26.454775 9209 ts_manager.cc:194] Registered new tserver with Master: d15c2de96cec44be9ed402dbaead6aba (127.2.70.195:39491)
I20260925 21:56:26.455122 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.445099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50586 (local address 127.2.70.254:43873)
0925 21:56:26.446423 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.446425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.446431 (+ 6us) server_negotiation.cc:407] Connection header received
0925 21:56:26.446450 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.446451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.446462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.446522 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.447494 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.448294 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.449312 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.449624 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.449674 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.451027 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.451029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.451029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.451037 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.453968 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.454300 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.454302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.454302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.454361 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.454546 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.454547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.454548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.454684 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.454763 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.454936 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.455027 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1298,"thread_start_us":73,"threads_started":1}
I20260925 21:56:26.455495 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.440844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.195:49991)
0925 21:56:26.441028 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.441074 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.441665 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.441903 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.441908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.442309 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:26.442509 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.442514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.444268 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.444269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.444968 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.444970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.445441 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.445441 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.445513 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.446198 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:26.446202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:26.449216 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:26.453948 (+ 4732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.453949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.453949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.454635 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.454887 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:26.454887 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:26.454888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:26.454948 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:26.455207 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:26.455208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:26.455246 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.455314 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.455410 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":3,"spinlock_wait_cycles":7552,"thread_start_us":104,"threads_started":1}
I20260925 21:56:26.455495 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.441167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:49991 (local address 127.2.70.254:43873)
0925 21:56:26.441214 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.441216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.441227 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:56:26.441783 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.441785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.441796 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.441826 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.443469 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.444179 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.445071 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.445334 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.445371 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.449300 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.449301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.449302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.449308 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.451579 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.454713 (+ 3134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.454714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.454715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.454775 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.455020 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.455021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.455021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.455130 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.455163 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.455355 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.455422 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 21:56:26.455785 9667 heartbeater.cc:347] Connected to a master server at 127.2.70.254:43873
I20260925 21:56:26.455858 9667 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:26.455927 9667 heartbeater.cc:511] Master 127.2.70.254:43873 requested a full tablet report, sending...
I20260925 21:56:26.456122 8428 ts_manager.cc:194] Re-registered known tserver with Master: d15c2de96cec44be9ed402dbaead6aba (127.2.70.195:39491)
I20260925 21:56:26.456549 8428 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:49991
I20260925 21:56:26.457486 8428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:26.459408 8428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:26.459503 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.445453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52556 (local address 127.2.70.253:43233)
0925 21:56:26.446173 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.446175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.446363 (+ 188us) server_negotiation.cc:407] Connection header received
0925 21:56:26.446388 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.446389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.446401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.446462 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.447245 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.454731 (+ 7486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.455846 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.456108 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.456166 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.457387 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.457388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.457388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.457394 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.458490 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.458809 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.458810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.458810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.458852 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.459046 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.459047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.459047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.459140 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.459207 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.459333 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.459413 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":58,"threads_started":1}
I20260925 21:56:26.466445 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.462836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:35501 (local address 127.0.0.1:54512)
0925 21:56:26.463049 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.463058 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.463098 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.463325 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.463326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.463328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:26.463372 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.463374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.464373 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.464374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.465316 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.465318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.466207 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.466208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.466264 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.466269 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.466319 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.466357 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":23,"thread_start_us":80,"threads_started":1}
I20260925 21:56:26.466549 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.462773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:43971 (local address 127.0.0.1:52630)
0925 21:56:26.462863 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.462874 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.462912 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.463462 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.463462 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.463464 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:26.463496 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.463498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.464783 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.464784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.465595 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.465598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.466355 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.466355 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.466414 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.466418 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.466443 (+ 25us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.466480 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 21:56:26.466742 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.463031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.2.70.194:35501)
0925 21:56:26.463180 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.463185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.463195 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:26.463231 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.463233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.463246 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.463319 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:26.463477 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.464301 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.465407 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.466139 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.466197 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.466271 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.466335 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.466376 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260925 21:56:26.466914 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.462879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.2.70.193:43971)
0925 21:56:26.463311 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.463316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.463324 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:26.463357 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.463359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.463372 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.463464 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:26.463547 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.464693 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.465682 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.466267 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.466698 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.466776 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.466800 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.466838 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":79,"threads_started":1}
I20260925 21:56:26.466995 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.462849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:39491 (local address 127.0.0.1:36380)
0925 21:56:26.463170 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.463178 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.463211 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.463447 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.463448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.463450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:26.463506 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.463508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.464869 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.464870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.465908 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.465910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.466744 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.466745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.466856 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.466860 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.466891 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.466925 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260925 21:56:26.467344 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.463054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.2.70.195:39491)
0925 21:56:26.463226 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.463231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.463241 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:26.463276 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.463278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.463292 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.463632 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:26.463783 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.464799 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.465987 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.466662 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.467139 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.467214 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.467237 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.467274 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260925 21:56:26.467445 9464 tablet_service.cc:1467] Processing CreateTablet for tablet bb3cf497350842eaa44381c9bfc27768 (DEFAULT_TABLE table=test-table [id=58eaa7b2abb0408c8b0f5e5497ba3ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:26.467790 9464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3cf497350842eaa44381c9bfc27768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:26.468066 9600 tablet_service.cc:1467] Processing CreateTablet for tablet bb3cf497350842eaa44381c9bfc27768 (DEFAULT_TABLE table=test-table [id=58eaa7b2abb0408c8b0f5e5497ba3ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:26.468382 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3cf497350842eaa44381c9bfc27768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:26.471336 9692 tablet_bootstrap.cc:492] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d: Bootstrap starting.
I20260925 21:56:26.471568 9693 tablet_bootstrap.cc:492] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba: Bootstrap starting.
I20260925 21:56:26.472033 9692 tablet_bootstrap.cc:654] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:26.472101 9327 tablet_service.cc:1467] Processing CreateTablet for tablet bb3cf497350842eaa44381c9bfc27768 (DEFAULT_TABLE table=test-table [id=58eaa7b2abb0408c8b0f5e5497ba3ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:26.472252 9693 tablet_bootstrap.cc:654] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:26.472327 9692 log.cc:824] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:26.472425 9327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3cf497350842eaa44381c9bfc27768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:26.473032 9693 log.cc:824] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:26.473160 9692 tablet_bootstrap.cc:492] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d: No bootstrap required, opened a new log
I20260925 21:56:26.473223 9692 ts_tablet_manager.cc:1397] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:26.473812 9693 tablet_bootstrap.cc:492] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba: No bootstrap required, opened a new log
I20260925 21:56:26.473878 9693 ts_tablet_manager.cc:1397] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:56:26.475262 9696 tablet_bootstrap.cc:492] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb: Bootstrap starting.
I20260925 21:56:26.475549 9692 raft_consensus.cc:348] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } }
I20260925 21:56:26.475736 9692 raft_consensus.cc:374] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:26.475785 9692 raft_consensus.cc:729] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be7ac9f2c9d4849b2f19f16a277448d, State: Initialized, Role: FOLLOWER
I20260925 21:56:26.475761 9693 raft_consensus.cc:348] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } }
I20260925 21:56:26.475924 9693 raft_consensus.cc:374] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:26.475960 9693 raft_consensus.cc:729] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15c2de96cec44be9ed402dbaead6aba, State: Initialized, Role: FOLLOWER
I20260925 21:56:26.475955 9692 consensus_queue.cc:260] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } }
I20260925 21:56:26.476012 9696 tablet_bootstrap.cc:654] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:26.476065 9693 consensus_queue.cc:260] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } }
I20260925 21:56:26.476202 9692 ts_tablet_manager.cc:1428] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:56:26.476279 9693 ts_tablet_manager.cc:1428] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 21:56:26.476503 9667 heartbeater.cc:503] Master 127.2.70.254:43873 was elected leader, sending a full tablet report...
I20260925 21:56:26.476521 9531 heartbeater.cc:503] Master 127.2.70.254:43873 was elected leader, sending a full tablet report...
I20260925 21:56:26.476310 9696 log.cc:824] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:26.478031 9696 tablet_bootstrap.cc:492] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb: No bootstrap required, opened a new log
I20260925 21:56:26.478087 9696 ts_tablet_manager.cc:1397] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 21:56:26.479470 9696 raft_consensus.cc:348] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } }
I20260925 21:56:26.479590 9696 raft_consensus.cc:374] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:26.479619 9696 raft_consensus.cc:729] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0882610850c9425ea77ae50c3a43dfbb, State: Initialized, Role: FOLLOWER
I20260925 21:56:26.479732 9696 consensus_queue.cc:260] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } }
I20260925 21:56:26.479998 9696 ts_tablet_manager.cc:1428] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 21:56:26.480137 9394 heartbeater.cc:503] Master 127.2.70.254:43873 was elected leader, sending a full tablet report...
I20260925 21:56:26.500167 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.497522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.2.70.252:45607)
0925 21:56:26.497574 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.497576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.497586 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:26.497632 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.497633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.497654 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.497712 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:26.497880 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.498494 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.499414 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.499868 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.499934 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.500021 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.500061 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.500099 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 21:56:26.500157 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.497436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:45607 (local address 127.0.0.1:51982)
0925 21:56:26.497496 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.497511 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.497580 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.497728 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.497729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.497731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:26.497787 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.497790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.498563 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.498564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.499324 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.499326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.499933 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.499933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.499975 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.499978 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.500055 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.500092 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 21:56:26.500526 9224 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260925 21:56:26.501389 9253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1de8b50ece3044268f93c293784dc39b. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.501508 9253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: This master's current role is: LEARNER
W20260925 21:56:26.557256 9533 tablet.cc:2369] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:26.596575 9697 raft_consensus.cc:482] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:26.596724 9697 raft_consensus.cc:504] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } }
I20260925 21:56:26.597107 9697 leader_election.cc:288] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0882610850c9425ea77ae50c3a43dfbb (127.2.70.193:43971), d15c2de96cec44be9ed402dbaead6aba (127.2.70.195:39491)
I20260925 21:56:26.601186 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.597318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:43971 (local address 127.2.70.194:54937)
0925 21:56:26.597387 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.597402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.597444 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.597703 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.597704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.597705 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:26.597818 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.597820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.598772 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.598772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.600135 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.600138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.600910 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.600911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.600995 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.600999 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.601055 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.601099 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":122}
I20260925 21:56:26.601205 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.597513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:54937 (local address 127.2.70.193:43971)
0925 21:56:26.597600 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.597601 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.597609 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:26.597632 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.597633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.597643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.597695 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:26.597908 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.598688 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.600235 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.600834 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.600926 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.600936 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.601079 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.601132 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 21:56:26.601205 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.597380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:52289 (local address 127.2.70.195:39491)
0925 21:56:26.597447 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.597449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.597573 (+ 124us) server_negotiation.cc:407] Connection header received
0925 21:56:26.597611 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.597612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.597626 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.597694 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:26.597908 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.598699 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.600236 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.600843 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.600927 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.600936 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.601108 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.601147 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 21:56:26.601194 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.597318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:39491 (local address 127.2.70.194:52289)
0925 21:56:26.597475 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:26.597487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:26.597557 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:26.597701 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:26.597702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:26.597703 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:26.597819 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.597820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.598771 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.598772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.600135 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:26.600138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.600926 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.600927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:26.601045 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.601048 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:26.601095 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 21:56:26.601121 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":176}
I20260925 21:56:26.601713 9347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3cf497350842eaa44381c9bfc27768" candidate_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882610850c9425ea77ae50c3a43dfbb" is_pre_election: true
I20260925 21:56:26.601879 9347 raft_consensus.cc:2454] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0be7ac9f2c9d4849b2f19f16a277448d in term 0.
I20260925 21:56:26.602101 9417 leader_election.cc:302] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0882610850c9425ea77ae50c3a43dfbb, 0be7ac9f2c9d4849b2f19f16a277448d; no voters:
I20260925 21:56:26.602106 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3cf497350842eaa44381c9bfc27768" candidate_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d15c2de96cec44be9ed402dbaead6aba" is_pre_election: true
I20260925 21:56:26.602234 9620 raft_consensus.cc:2454] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0be7ac9f2c9d4849b2f19f16a277448d in term 0.
I20260925 21:56:26.602236 9697 raft_consensus.cc:2781] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:26.602303 9697 raft_consensus.cc:482] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:26.602335 9697 raft_consensus.cc:3037] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:26.603494 9697 raft_consensus.cc:504] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } }
I20260925 21:56:26.603615 9697 leader_election.cc:288] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [CANDIDATE]: Term 1 election: Requested vote from peers 0882610850c9425ea77ae50c3a43dfbb (127.2.70.193:43971), d15c2de96cec44be9ed402dbaead6aba (127.2.70.195:39491)
I20260925 21:56:26.603755 9347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3cf497350842eaa44381c9bfc27768" candidate_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882610850c9425ea77ae50c3a43dfbb"
I20260925 21:56:26.603775 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3cf497350842eaa44381c9bfc27768" candidate_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d15c2de96cec44be9ed402dbaead6aba"
I20260925 21:56:26.603879 9620 raft_consensus.cc:3037] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:26.603866 9347 raft_consensus.cc:3037] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:26.604737 9620 raft_consensus.cc:2454] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0be7ac9f2c9d4849b2f19f16a277448d in term 1.
I20260925 21:56:26.604737 9347 raft_consensus.cc:2454] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0be7ac9f2c9d4849b2f19f16a277448d in term 1.
I20260925 21:56:26.604926 9417 leader_election.cc:302] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0882610850c9425ea77ae50c3a43dfbb, 0be7ac9f2c9d4849b2f19f16a277448d; no voters:
I20260925 21:56:26.605072 9697 raft_consensus.cc:2781] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:26.605221 9697 raft_consensus.cc:686] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 1 LEADER]: Becoming Leader. State: Replica: 0be7ac9f2c9d4849b2f19f16a277448d, State: Running, Role: LEADER
I20260925 21:56:26.605304 9697 consensus_queue.cc:237] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } }
I20260925 21:56:26.606140 8428 catalog_manager.cc:5495] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0be7ac9f2c9d4849b2f19f16a277448d (127.2.70.194). New cstate: current_term: 1 leader_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } health_report { overall_health: UNKNOWN } } }
I20260925 21:56:26.606703 9177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: NON_VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: true } } }
I20260925 21:56:26.606789 9177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:26.607540 9177 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b: attempting to promote NON_VOTER 29d6d14e921f4ef1b54a4dc181f1c577 to VOTER
I20260925 21:56:26.607661 9177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } }
I20260925 21:56:26.608127 9037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 FOLLOWER]: Refusing update from remote peer 1de8b50ece3044268f93c293784dc39b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 21:56:26.608147 9224 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 LEARNER]: Refusing update from remote peer 1de8b50ece3044268f93c293784dc39b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 21:56:26.608384 9178 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 21:56:26.608493 9708 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 21:56:26.608949 9178 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 29d6d14e921f4ef1b54a4dc181f1c577 (127.2.70.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } } }
I20260925 21:56:26.609252 9224 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 29d6d14e921f4ef1b54a4dc181f1c577 (127.2.70.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } } }
I20260925 21:56:26.609800 9178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } } }
I20260925 21:56:26.609831 9708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } } }
I20260925 21:56:26.609884 9708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:26.609964 9707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } } }
I20260925 21:56:26.609874 9178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:26.610020 9707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:26.610042 9037 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 29d6d14e921f4ef1b54a4dc181f1c577 (127.2.70.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } } }
I20260925 21:56:26.610100 9253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } } }
I20260925 21:56:26.610160 9253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 21:56:26.610875 9710 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1de8b50ece3044268f93c293784dc39b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } } }
I20260925 21:56:26.611016 9710 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 21:56:26.647709 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.645333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54524 (local address 127.2.70.194:35501)
0925 21:56:26.645383 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.645385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.645394 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:26.645436 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.645437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.645461 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.645523 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:26.645677 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.646213 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.646970 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.647193 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.647249 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.647399 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:26.647502 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:26.647570 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.647607 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.647630 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 21:56:26.649500 9620 raft_consensus.cc:1261] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba [term 1 FOLLOWER]: Refusing update from remote peer 0be7ac9f2c9d4849b2f19f16a277448d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:26.649827 9347 raft_consensus.cc:1261] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb [term 1 FOLLOWER]: Refusing update from remote peer 0be7ac9f2c9d4849b2f19f16a277448d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:26.649855 9705 consensus_queue.cc:1035] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:26.650144 9705 consensus_queue.cc:1035] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:26.652817 9715 mvcc.cc:205] Tried to move back new op lower bound from 7333369391713918976 to 7333369391535915008. Current Snapshot: MvccSnapshot[applied={T|T < 7333369391713918976}]
I20260925 21:56:26.653040 9712 mvcc.cc:205] Tried to move back new op lower bound from 7333369391713918976 to 7333369391535915008. Current Snapshot: MvccSnapshot[applied={T|T < 7333369391713918976}]
I20260925 21:56:26.654397 9713 mvcc.cc:205] Tried to move back new op lower bound from 7333369391713918976 to 7333369391535915008. Current Snapshot: MvccSnapshot[applied={T|T < 7333369391713918976}]
I20260925 21:56:26.656291 8427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:26.656388 8427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:26.657775 8427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1de8b50ece3044268f93c293784dc39b: Sending DeleteTablet for 3 replicas of tablet bb3cf497350842eaa44381c9bfc27768
I20260925 21:56:26.658114 9600 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3cf497350842eaa44381c9bfc27768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:26 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:36380
I20260925 21:56:26.658313 9721 tablet_replica.cc:330] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba: stopping tablet replica
I20260925 21:56:26.658406 9721 raft_consensus.cc:2229] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:26.658514 9721 raft_consensus.cc:2258] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:26.658983 9327 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3cf497350842eaa44381c9bfc27768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:26 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:52630
I20260925 21:56:26.659184 9721 ts_tablet_manager.cc:1905] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:26.659375 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 8397
I20260925 21:56:26.660971 9721 ts_tablet_manager.cc:1918] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:26.661034 9721 log.cc:1196] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/bb3cf497350842eaa44381c9bfc27768
I20260925 21:56:26.661293 9721 ts_tablet_manager.cc:1939] T bb3cf497350842eaa44381c9bfc27768 P d15c2de96cec44be9ed402dbaead6aba: Deleting consensus metadata
I20260925 21:56:26.661700 9464 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3cf497350842eaa44381c9bfc27768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:26 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:54512
I20260925 21:56:26.661899 9724 tablet_replica.cc:330] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d: stopping tablet replica
I20260925 21:56:26.661991 9724 raft_consensus.cc:2229] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:26.662110 9724 raft_consensus.cc:2258] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:26.662829 9724 ts_tablet_manager.cc:1905] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:26.664965 9724 ts_tablet_manager.cc:1918] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:26.665027 9724 log.cc:1196] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/bb3cf497350842eaa44381c9bfc27768
I20260925 21:56:26.665269 9724 ts_tablet_manager.cc:1939] T bb3cf497350842eaa44381c9bfc27768 P 0be7ac9f2c9d4849b2f19f16a277448d: Deleting consensus metadata
W20260925 21:56:26.666529 9417 connection.cc:535] server connection from 127.0.0.1:54512 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 21:56:26.666563 9555 connection.cc:535] server connection from 127.0.0.1:36380 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 21:56:26.666759 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.658607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.2.70.252:45607)
0925 21:56:26.658710 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.658712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.658829 (+ 117us) server_negotiation.cc:407] Connection header received
0925 21:56:26.658919 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.658920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.658929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.658957 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.659715 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.660527 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.661690 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.662017 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.662062 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.663723 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.663725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.663725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.663732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.665416 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.665805 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.665806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.665807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.665876 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.666086 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.666087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.666087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.666221 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.666290 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.666578 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.666663 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 21:56:26.667639 9723 tablet_replica.cc:330] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb: stopping tablet replica
I20260925 21:56:26.667793 9723 raft_consensus.cc:2229] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:26.667963 9723 raft_consensus.cc:2258] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:26.668511 9723 ts_tablet_manager.cc:1905] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:26.670077 9723 ts_tablet_manager.cc:1918] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:26.670168 9723 log.cc:1196] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-0/wal/wals/bb3cf497350842eaa44381c9bfc27768
I20260925 21:56:26.670441 9723 ts_tablet_manager.cc:1939] T bb3cf497350842eaa44381c9bfc27768 P 0882610850c9425ea77ae50c3a43dfbb: Deleting consensus metadata
W20260925 21:56:26.670878 9281 connection.cc:441] server connection from 127.0.0.1:52630 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52630 (request call id 1) could send its response
I20260925 21:56:26.677629 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.670032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.2.70.253:43233)
0925 21:56:26.670104 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.670106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.671104 (+ 998us) server_negotiation.cc:407] Connection header received
0925 21:56:26.671204 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.671206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.671220 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.671282 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.671953 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.672453 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.673303 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.673623 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.673684 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.674975 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.674976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.674977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.674984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.676482 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.676840 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.676841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.676841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.676899 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.677119 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.677120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.677121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.677272 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.677360 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.677453 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.677534 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 21:56:26.677919 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:26.670132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.2.70.252:45607)
0925 21:56:26.670171 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:26.670172 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:26.670341 (+ 169us) server_negotiation.cc:407] Connection header received
0925 21:56:26.670386 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:26.670387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:26.670397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:26.670453 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:26.671371 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.671928 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.673004 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:26.673315 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:26.673367 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:26.674933 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:26.674934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:26.674934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:26.674941 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:26.676524 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.676957 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.676958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.676958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.677011 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:26.677305 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:26.677307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:26.677307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:26.677433 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:26.677500 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:26.677729 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 21:56:26.677808 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 21:56:27.654176 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:27.653838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.193:42333)
0925 21:56:27.653984 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:27.654119 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260925 21:56:27.654472 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:27.654129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.194:40173)
0925 21:56:27.654270 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:27.654425 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":45,"threads_started":1}
W20260925 21:56:27.654543 9394 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:43873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
I20260925 21:56:27.654814 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:27.654611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.195:35959)
0925 21:56:27.654709 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:27.654774 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
W20260925 21:56:27.654827 9531 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:43873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
W20260925 21:56:27.655090 9667 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:43873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
I20260925 21:56:28.158659 9734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1de8b50ece3044268f93c293784dc39b)
I20260925 21:56:28.158726 9735 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1de8b50ece3044268f93c293784dc39b)
I20260925 21:56:28.158790 9734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } }
I20260925 21:56:28.158840 9735 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } }
I20260925 21:56:28.159051 9734 leader_election.cc:288] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873), f0b5e1ad65054f35a797c5a6b0e45ed6 (127.2.70.253:43233)
I20260925 21:56:28.159410 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.159164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51050)
0925 21:56:28.159299 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.159347 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":55,"threads_started":1}
I20260925 21:56:28.159402 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.159154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51034)
0925 21:56:28.159317 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.159361 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":51,"threads_started":1}
W20260925 21:56:28.159551 9011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
W20260925 21:56:28.159696 9198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
I20260925 21:56:28.159740 9735 leader_election.cc:288] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873), 29d6d14e921f4ef1b54a4dc181f1c577 (127.2.70.252:45607)
I20260925 21:56:28.160051 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.159950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51064)
0925 21:56:28.159987 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.160019 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 21:56:28.160113 9011 leader_election.cc:334] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873): Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
I20260925 21:56:28.160261 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.160096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51078)
0925 21:56:28.160214 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.160239 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260925 21:56:28.160326 9198 leader_election.cc:334] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873): Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
I20260925 21:56:28.163185 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.159191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43233 (local address 127.0.0.1:52562)
0925 21:56:28.159347 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.159357 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:28.159412 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:28.159733 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:28.159734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:28.159735 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:28.159819 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:28.159821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:28.160835 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:28.160835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:28.162112 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:28.162116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:28.162900 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:28.162901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:28.162970 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:28.162975 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:56:28.163030 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 21:56:28.163071 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":47,"threads_started":1}
I20260925 21:56:28.163174 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.159340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:45607 (local address 127.0.0.1:52006)
0925 21:56:28.159479 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.159487 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:28.159531 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:28.159768 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:28.159769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:28.159771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:28.159848 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:28.159850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:28.160835 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:28.160835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:28.162112 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:28.162115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:28.162900 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:28.162901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:28.162970 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:28.162975 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:56:28.163030 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 21:56:28.163075 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":37,"threads_started":1}
I20260925 21:56:28.163230 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.159513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.2.70.252:45607)
0925 21:56:28.159637 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:28.159638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:28.159644 (+ 6us) server_negotiation.cc:407] Connection header received
0925 21:56:28.159664 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:28.159664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:28.159672 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:28.159729 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:28.160011 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:28.160735 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:28.162216 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:28.162805 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:28.162883 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:28.162892 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:28.163067 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 21:56:28.163113 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":237,"server-negotiator.queue_time_us":111,"thread_start_us":44,"threads_started":1}
I20260925 21:56:28.163331 9739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.159240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.2.70.253:43233)
0925 21:56:28.159604 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:28.159605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:28.159611 (+ 6us) server_negotiation.cc:407] Connection header received
0925 21:56:28.159630 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:28.159631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:28.159640 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:28.159913 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:28.159957 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:28.160747 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:28.162232 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:28.162818 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:28.162877 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:28.162891 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:28.163067 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 21:56:28.163114 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":53,"threads_started":1}
I20260925 21:56:28.163512 9037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" is_pre_election: true
I20260925 21:56:28.163501 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" is_pre_election: true
I20260925 21:56:28.163602 9224 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0b5e1ad65054f35a797c5a6b0e45ed6 in term 1.
I20260925 21:56:28.163604 9037 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d6d14e921f4ef1b54a4dc181f1c577 in term 1.
I20260925 21:56:28.163734 9199 leader_election.cc:302] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29d6d14e921f4ef1b54a4dc181f1c577, f0b5e1ad65054f35a797c5a6b0e45ed6; no voters: 1de8b50ece3044268f93c293784dc39b
I20260925 21:56:28.163763 9011 leader_election.cc:302] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29d6d14e921f4ef1b54a4dc181f1c577, f0b5e1ad65054f35a797c5a6b0e45ed6; no voters: 1de8b50ece3044268f93c293784dc39b
I20260925 21:56:28.163856 9734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 21:56:28.163882 9735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 21:56:28.163897 9734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1de8b50ece3044268f93c293784dc39b)
I20260925 21:56:28.163908 9735 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1de8b50ece3044268f93c293784dc39b)
I20260925 21:56:28.163929 9734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 1 FOLLOWER]: Advancing to term 2
I20260925 21:56:28.163928 9735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 1 FOLLOWER]: Advancing to term 2
I20260925 21:56:28.164764 9735 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } }
I20260925 21:56:28.164773 9734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } }
I20260925 21:56:28.164907 9734 leader_election.cc:288] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [CANDIDATE]: Term 2 election: Requested vote from peers 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873), f0b5e1ad65054f35a797c5a6b0e45ed6 (127.2.70.253:43233)
I20260925 21:56:28.165050 9037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6"
I20260925 21:56:28.165132 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.165042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51090)
0925 21:56:28.165073 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.165107 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 21:56:28.165136 9037 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 29d6d14e921f4ef1b54a4dc181f1c577 in current term 2: Already voted for candidate f0b5e1ad65054f35a797c5a6b0e45ed6 in this term.
I20260925 21:56:28.165246 9735 leader_election.cc:288] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [CANDIDATE]: Term 2 election: Requested vote from peers 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873), 29d6d14e921f4ef1b54a4dc181f1c577 (127.2.70.252:45607)
I20260925 21:56:28.165375 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.165149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51096)
0925 21:56:28.165326 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.165352 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260925 21:56:28.165445 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.165355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51108)
0925 21:56:28.165386 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.165413 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 21:56:28.165472 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "29d6d14e921f4ef1b54a4dc181f1c577"
I20260925 21:56:28.165544 9224 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0b5e1ad65054f35a797c5a6b0e45ed6 in current term 2: Already voted for candidate 29d6d14e921f4ef1b54a4dc181f1c577 in this term.
W20260925 21:56:28.165593 9011 leader_election.cc:334] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873): Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
I20260925 21:56:28.165643 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.165557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51110)
0925 21:56:28.165599 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.165622 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 21:56:28.165664 9011 leader_election.cc:302] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0b5e1ad65054f35a797c5a6b0e45ed6; no voters: 1de8b50ece3044268f93c293784dc39b, 29d6d14e921f4ef1b54a4dc181f1c577
W20260925 21:56:28.165710 9198 leader_election.cc:334] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873): Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
I20260925 21:56:28.165750 9198 leader_election.cc:302] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d6d14e921f4ef1b54a4dc181f1c577; no voters: 1de8b50ece3044268f93c293784dc39b, f0b5e1ad65054f35a797c5a6b0e45ed6
I20260925 21:56:28.165776 9735 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 21:56:28.165843 9734 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 21:56:28.655270 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.654859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.193:55311)
0925 21:56:28.655050 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.655164 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":63,"threads_started":1}
I20260925 21:56:28.655611 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.655330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.195:45585)
0925 21:56:28.655536 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.655571 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":53,"threads_started":1}
I20260925 21:56:28.655699 9745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:28.655161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.194:45211)
0925 21:56:28.655632 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:28.655663 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":63,"threads_started":1}
I20260925 21:56:29.656117 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.655807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.193:59427)
0925 21:56:29.656022 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.656064 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
I20260925 21:56:29.656551 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.656071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.195:33185)
0925 21:56:29.656466 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.656502 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":50,"threads_started":1}
I20260925 21:56:29.656598 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.656071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.194:43441)
0925 21:56:29.656515 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.656551 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":52,"threads_started":1}
I20260925 21:56:29.790733 9750 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:29.790853 9750 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } }
I20260925 21:56:29.791028 9750 leader_election.cc:288] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873), 29d6d14e921f4ef1b54a4dc181f1c577 (127.2.70.252:45607)
I20260925 21:56:29.791391 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.791209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51346)
0925 21:56:29.791306 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.791346 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":41,"threads_started":1}
I20260925 21:56:29.791486 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" is_pre_election: true
I20260925 21:56:29.791590 9224 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0b5e1ad65054f35a797c5a6b0e45ed6 in term 2.
I20260925 21:56:29.791730 9011 leader_election.cc:302] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d6d14e921f4ef1b54a4dc181f1c577, f0b5e1ad65054f35a797c5a6b0e45ed6; no voters:
I20260925 21:56:29.791816 9750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 21:56:29.791903 9750 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:29.791921 9750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 2 FOLLOWER]: Advancing to term 3
I20260925 21:56:29.791800 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.791684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51356)
0925 21:56:29.791740 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.791774 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 21:56:29.792081 9011 leader_election.cc:334] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873): Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
I20260925 21:56:29.792661 9750 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } }
I20260925 21:56:29.792763 9750 leader_election.cc:288] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [CANDIDATE]: Term 3 election: Requested vote from peers 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873), 29d6d14e921f4ef1b54a4dc181f1c577 (127.2.70.252:45607)
I20260925 21:56:29.792985 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.792873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51360)
0925 21:56:29.792921 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.792959 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 21:56:29.793099 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "29d6d14e921f4ef1b54a4dc181f1c577"
I20260925 21:56:29.793188 9224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 2 FOLLOWER]: Advancing to term 3
I20260925 21:56:29.793344 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.793275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51370)
0925 21:56:29.793304 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.793329 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 21:56:29.793421 9011 leader_election.cc:334] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873): Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
I20260925 21:56:29.793973 9224 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0b5e1ad65054f35a797c5a6b0e45ed6 in term 3.
I20260925 21:56:29.794152 9011 leader_election.cc:302] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29d6d14e921f4ef1b54a4dc181f1c577, f0b5e1ad65054f35a797c5a6b0e45ed6; no voters: 1de8b50ece3044268f93c293784dc39b
I20260925 21:56:29.794232 9750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 21:56:29.794332 9750 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [term 3 LEADER]: Becoming Leader. State: Replica: f0b5e1ad65054f35a797c5a6b0e45ed6, State: Running, Role: LEADER
I20260925 21:56:29.794406 9750 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } }
I20260925 21:56:29.794751 9753 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0b5e1ad65054f35a797c5a6b0e45ed6. Latest consensus state: current_term: 3 leader_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } } }
I20260925 21:56:29.794847 9753 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:29.795017 9755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:29.795452 9755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:29.795513 9755 catalog_manager.cc:1259] Loaded cluster ID: b18e6ffcbfb144a8a82a34aecb3db065
I20260925 21:56:29.795539 9755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:29.795712 9755 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:29.795782 9755 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: Loaded TSK: 0
I20260925 21:56:29.795948 9755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:56:29.839665 9027 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:29.841486 9027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:29.842590 9224 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [term 3 FOLLOWER]: Refusing update from remote peer f0b5e1ad65054f35a797c5a6b0e45ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 21:56:29.842638 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.842484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51396)
0925 21:56:29.842546 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.842594 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 21:56:29.842828 9753 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 21:56:29.842986 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.842880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.0.0.1:51400)
0925 21:56:29.842931 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.842962 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 21:56:29.843055 9011 consensus_peers.cc:487] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 -> Peer 1de8b50ece3044268f93c293784dc39b (127.2.70.254:43873): Couldn't send request to peer 1de8b50ece3044268f93c293784dc39b. Status: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 21:56:29.843451 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0b5e1ad65054f35a797c5a6b0e45ed6. Latest consensus state: current_term: 3 leader_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } } }
I20260925 21:56:29.843554 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 21:56:29.843952 9750 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } } }
I20260925 21:56:29.844062 9750 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:29.844116 9758 mvcc.cc:205] Tried to move back new op lower bound from 7333369404793028608 to 7333369404598419456. Current Snapshot: MvccSnapshot[applied={T|T < 7333369404793028608}]
I20260925 21:56:29.844260 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1de8b50ece3044268f93c293784dc39b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43873 } } peers { permanent_uuid: "f0b5e1ad65054f35a797c5a6b0e45ed6" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43233 } attrs { promote: false } } peers { permanent_uuid: "29d6d14e921f4ef1b54a4dc181f1c577" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 45607 } attrs { promote: false } } }
I20260925 21:56:29.844465 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d6d14e921f4ef1b54a4dc181f1c577 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 21:56:29.849138 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.845867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:35501 (local address 127.0.0.1:54528)
0925 21:56:29.846159 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.846168 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:29.846205 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:29.846534 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:29.846535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:29.846537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:29.846567 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:29.846568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.847376 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.847377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:29.848165 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:29.848167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.848890 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.848891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:29.848953 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:29.848964 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 21:56:29.849020 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 21:56:29.849061 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":78,"threads_started":1}
I20260925 21:56:29.849146 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.846018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54528 (local address 127.2.70.194:35501)
0925 21:56:29.846248 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:29.846249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:29.846256 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:56:29.846277 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:29.846278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:29.846288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:29.846353 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:29.846622 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.847301 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.848266 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.848807 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.848891 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:29.848905 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:29.849033 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 21:56:29.849075 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":53,"threads_started":1}
I20260925 21:56:29.849282 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.846025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:43971 (local address 127.0.0.1:52642)
0925 21:56:29.846222 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.846232 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:29.846311 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:29.846484 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:29.846485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:29.846486 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:29.846534 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:29.846535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.847270 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.847270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:29.848457 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:29.848459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.849079 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.849080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:29.849134 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:29.849137 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:29.849177 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 21:56:29.849211 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":61,"thread_start_us":54,"threads_started":1}
I20260925 21:56:29.849375 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.845827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:39491 (local address 127.0.0.1:36394)
0925 21:56:29.845863 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.845874 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:29.845919 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:29.846096 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:29.846097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:29.846098 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:29.846152 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:29.846154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.847224 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.847225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:29.848454 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:29.848456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.849167 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.849169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:29.849228 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:29.849233 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:56:29.849260 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 21:56:29.849304 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":6}
I20260925 21:56:29.849480 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.846121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.2.70.193:43971)
0925 21:56:29.846226 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:29.846228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:29.846282 (+ 54us) server_negotiation.cc:407] Connection header received
0925 21:56:29.846414 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:29.846415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:29.846426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:29.846485 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:29.846606 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.847164 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.848546 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.848976 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.849040 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:29.849052 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:29.849359 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 21:56:29.849404 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":48,"threads_started":1}
I20260925 21:56:29.849520 9464 tablet_service.cc:1467] Processing CreateTablet for tablet ff7fc0d7664c430b97e75744e78185c2 (DEFAULT_TABLE table=test-table [id=f2a581e87c3b4e2e8a0794c09d415c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:29.849648 9464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7fc0d7664c430b97e75744e78185c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:29.850041 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.845895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.2.70.195:39491)
0925 21:56:29.845988 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:29.845990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:29.845998 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:29.846021 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:29.846022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:29.846032 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:29.846096 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:29.846262 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.847149 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.848539 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.849081 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.849907 (+ 826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:29.849922 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:29.849940 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 21:56:29.849979 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":46,"threads_started":1}
I20260925 21:56:29.850252 9600 tablet_service.cc:1467] Processing CreateTablet for tablet ff7fc0d7664c430b97e75744e78185c2 (DEFAULT_TABLE table=test-table [id=f2a581e87c3b4e2e8a0794c09d415c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:29.850373 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7fc0d7664c430b97e75744e78185c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:29.851110 9768 tablet_bootstrap.cc:492] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d: Bootstrap starting.
I20260925 21:56:29.851230 9327 tablet_service.cc:1467] Processing CreateTablet for tablet ff7fc0d7664c430b97e75744e78185c2 (DEFAULT_TABLE table=test-table [id=f2a581e87c3b4e2e8a0794c09d415c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:29.851334 9327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7fc0d7664c430b97e75744e78185c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:29.851480 9769 tablet_bootstrap.cc:492] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba: Bootstrap starting.
I20260925 21:56:29.851521 9768 tablet_bootstrap.cc:654] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:29.851795 9769 tablet_bootstrap.cc:654] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:29.852125 9768 tablet_bootstrap.cc:492] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d: No bootstrap required, opened a new log
I20260925 21:56:29.852175 9768 ts_tablet_manager.cc:1397] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:29.852352 9769 tablet_bootstrap.cc:492] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba: No bootstrap required, opened a new log
I20260925 21:56:29.852312 9768 raft_consensus.cc:348] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } }
I20260925 21:56:29.852387 9768 raft_consensus.cc:374] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:29.852401 9769 ts_tablet_manager.cc:1397] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:29.852407 9768 raft_consensus.cc:729] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be7ac9f2c9d4849b2f19f16a277448d, State: Initialized, Role: FOLLOWER
I20260925 21:56:29.852471 9768 consensus_queue.cc:260] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } }
I20260925 21:56:29.852510 9769 raft_consensus.cc:348] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } }
I20260925 21:56:29.852595 9769 raft_consensus.cc:374] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:29.852627 9769 raft_consensus.cc:729] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15c2de96cec44be9ed402dbaead6aba, State: Initialized, Role: FOLLOWER
I20260925 21:56:29.852671 9768 ts_tablet_manager.cc:1428] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:29.852703 9769 consensus_queue.cc:260] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } }
I20260925 21:56:29.852789 9773 tablet_bootstrap.cc:492] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb: Bootstrap starting.
I20260925 21:56:29.852885 9769 ts_tablet_manager.cc:1428] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:29.853159 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.853057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.194:60407)
0925 21:56:29.853096 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.853130 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 21:56:29.853165 9773 tablet_bootstrap.cc:654] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:29.853421 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.853265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.195:50981)
0925 21:56:29.853372 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.853397 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 21:56:29.853955 9532 heartbeater.cc:503] Master 127.2.70.253:43233 was elected leader, sending a full tablet report...
I20260925 21:56:29.854022 9773 tablet_bootstrap.cc:492] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb: No bootstrap required, opened a new log
I20260925 21:56:29.854065 9773 ts_tablet_manager.cc:1397] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:29.854125 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.854038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.194:57665)
0925 21:56:29.854076 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.854100 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 21:56:29.854162 9773 raft_consensus.cc:348] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } }
I20260925 21:56:29.854221 9773 raft_consensus.cc:374] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:29.854239 9773 raft_consensus.cc:729] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0882610850c9425ea77ae50c3a43dfbb, State: Initialized, Role: FOLLOWER
I20260925 21:56:29.854295 9773 consensus_queue.cc:260] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } }
I20260925 21:56:29.854403 9668 heartbeater.cc:503] Master 127.2.70.253:43233 was elected leader, sending a full tablet report...
I20260925 21:56:29.854501 9773 ts_tablet_manager.cc:1428] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:29.854583 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.854498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.195:37557)
0925 21:56:29.854534 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.854562 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 21:56:29.854902 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.854768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.193:35053)
0925 21:56:29.854838 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.854875 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 21:56:29.855124 9395 heartbeater.cc:503] Master 127.2.70.253:43233 was elected leader, sending a full tablet report...
W20260925 21:56:29.918882 9396 tablet.cc:2369] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:29.936354 9774 raft_consensus.cc:482] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:29.936450 9774 raft_consensus.cc:504] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } }
I20260925 21:56:29.936779 9774 leader_election.cc:288] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0be7ac9f2c9d4849b2f19f16a277448d (127.2.70.194:35501), 0882610850c9425ea77ae50c3a43dfbb (127.2.70.193:43971)
I20260925 21:56:29.940753 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.937058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:43303 (local address 127.2.70.194:35501)
0925 21:56:29.937105 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:29.937107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:29.937151 (+ 44us) server_negotiation.cc:407] Connection header received
0925 21:56:29.937180 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:29.937181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:29.937192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:29.937261 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:29.937420 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.938209 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.939746 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.940340 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.940415 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:29.940425 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:29.940635 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 21:56:29.940682 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 21:56:29.940753 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.937106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:45267 (local address 127.2.70.193:43971)
0925 21:56:29.937144 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:29.937145 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:29.937282 (+ 137us) server_negotiation.cc:407] Connection header received
0925 21:56:29.937306 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:29.937307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:29.937318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:29.937359 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:29.937486 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.938208 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.939762 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.940340 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.940414 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:29.940425 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:29.940634 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 21:56:29.940679 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 21:56:29.940763 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.936939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:35501 (local address 127.2.70.195:43303)
0925 21:56:29.937009 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.937022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:29.937087 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:29.937277 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:29.937277 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:29.937279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:29.937353 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:29.937355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.938301 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.938303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:29.939639 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:29.939642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.940442 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.940443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:29.940544 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:29.940552 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 21:56:29.940614 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 21:56:29.940670 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":69}
I20260925 21:56:29.940788 9777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.937037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:43971 (local address 127.2.70.195:45267)
0925 21:56:29.937179 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.937187 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:29.937227 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:29.937387 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:29.937388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:29.937389 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:29.937431 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:29.937432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.938302 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.938303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:29.939639 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:29.939642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.940438 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.940439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:29.940545 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:29.940550 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:56:29.940614 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 21:56:29.940670 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":68,"thread_start_us":104,"threads_started":1}
I20260925 21:56:29.941062 9484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7fc0d7664c430b97e75744e78185c2" candidate_uuid: "d15c2de96cec44be9ed402dbaead6aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" is_pre_election: true
I20260925 21:56:29.941205 9484 raft_consensus.cc:2454] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15c2de96cec44be9ed402dbaead6aba in term 0.
I20260925 21:56:29.941330 9347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7fc0d7664c430b97e75744e78185c2" candidate_uuid: "d15c2de96cec44be9ed402dbaead6aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882610850c9425ea77ae50c3a43dfbb" is_pre_election: true
I20260925 21:56:29.941412 9347 raft_consensus.cc:2454] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15c2de96cec44be9ed402dbaead6aba in term 0.
I20260925 21:56:29.941437 9553 leader_election.cc:302] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be7ac9f2c9d4849b2f19f16a277448d, d15c2de96cec44be9ed402dbaead6aba; no voters:
I20260925 21:56:29.941567 9774 raft_consensus.cc:2781] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:29.941605 9774 raft_consensus.cc:482] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:29.941629 9774 raft_consensus.cc:3037] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:29.942279 9774 raft_consensus.cc:504] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } }
I20260925 21:56:29.942389 9774 leader_election.cc:288] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [CANDIDATE]: Term 1 election: Requested vote from peers 0be7ac9f2c9d4849b2f19f16a277448d (127.2.70.194:35501), 0882610850c9425ea77ae50c3a43dfbb (127.2.70.193:43971)
I20260925 21:56:29.942556 9484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7fc0d7664c430b97e75744e78185c2" candidate_uuid: "d15c2de96cec44be9ed402dbaead6aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be7ac9f2c9d4849b2f19f16a277448d"
I20260925 21:56:29.942574 9347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7fc0d7664c430b97e75744e78185c2" candidate_uuid: "d15c2de96cec44be9ed402dbaead6aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882610850c9425ea77ae50c3a43dfbb"
I20260925 21:56:29.942637 9484 raft_consensus.cc:3037] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:29.942662 9347 raft_consensus.cc:3037] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:29.943240 9347 raft_consensus.cc:2454] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15c2de96cec44be9ed402dbaead6aba in term 1.
I20260925 21:56:29.943331 9484 raft_consensus.cc:2454] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15c2de96cec44be9ed402dbaead6aba in term 1.
W20260925 21:56:29.943413 9669 tablet.cc:2369] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:29.943509 9553 leader_election.cc:302] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be7ac9f2c9d4849b2f19f16a277448d, d15c2de96cec44be9ed402dbaead6aba; no voters:
I20260925 21:56:29.943602 9774 raft_consensus.cc:2781] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:29.943768 9774 raft_consensus.cc:686] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 1 LEADER]: Becoming Leader. State: Replica: d15c2de96cec44be9ed402dbaead6aba, State: Running, Role: LEADER
I20260925 21:56:29.943854 9774 consensus_queue.cc:237] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } } peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } }
I20260925 21:56:29.944113 9777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.943997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.195:47549)
0925 21:56:29.944050 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.944080 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 21:56:29.944641 9027 catalog_manager.cc:5495] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba reported cstate change: term changed from 0 to 1, leader changed from <none> to d15c2de96cec44be9ed402dbaead6aba (127.2.70.195). New cstate: current_term: 1 leader_uuid: "d15c2de96cec44be9ed402dbaead6aba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15c2de96cec44be9ed402dbaead6aba" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 } health_report { overall_health: UNKNOWN } } }
I20260925 21:56:29.965065 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.962809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.2.70.195:39491)
0925 21:56:29.962899 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:29.962901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:29.962909 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:29.962941 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:29.962942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:29.962961 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:29.963017 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:29.963117 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.963588 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.964329 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:29.964538 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:29.964603 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:29.964704 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:29.964808 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:29.964873 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:29.964963 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 21:56:29.964984 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 21:56:29.966189 9347 raft_consensus.cc:1261] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb [term 1 FOLLOWER]: Refusing update from remote peer d15c2de96cec44be9ed402dbaead6aba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:29.966209 9484 raft_consensus.cc:1261] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d [term 1 FOLLOWER]: Refusing update from remote peer d15c2de96cec44be9ed402dbaead6aba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:29.966478 9774 consensus_queue.cc:1035] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0882610850c9425ea77ae50c3a43dfbb" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:29.966653 9778 consensus_queue.cc:1035] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be7ac9f2c9d4849b2f19f16a277448d" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:29.967234 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.967109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.193:59255)
0925 21:56:29.967164 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.967198 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 21:56:29.968050 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.967920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.194:56345)
0925 21:56:29.967991 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.968016 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 21:56:29.968972 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.968829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.194:47083)
0925 21:56:29.968908 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.968946 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 21:56:29.969089 9777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.968975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.195:52611)
0925 21:56:29.969031 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.969061 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 21:56:29.969383 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.969259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43873 (local address 127.2.70.193:59115)
0925 21:56:29.969321 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.969345 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 21:56:29.971282 9027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:29.971362 9027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:29.972723 9027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0b5e1ad65054f35a797c5a6b0e45ed6: Sending DeleteTablet for 3 replicas of tablet ff7fc0d7664c430b97e75744e78185c2
I20260925 21:56:29.973174 9327 tablet_service.cc:1514] Processing DeleteTablet for tablet ff7fc0d7664c430b97e75744e78185c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:29 UTC) from {username='oryx', principal='oryx/127.2.70.253@KRBTEST.COM'} at 127.0.0.1:52642
I20260925 21:56:29.973218 9464 tablet_service.cc:1514] Processing DeleteTablet for tablet ff7fc0d7664c430b97e75744e78185c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:29 UTC) from {username='oryx', principal='oryx/127.2.70.253@KRBTEST.COM'} at 127.0.0.1:54528
I20260925 21:56:29.973433 9791 tablet_replica.cc:330] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb: stopping tablet replica
I20260925 21:56:29.973433 9600 tablet_service.cc:1514] Processing DeleteTablet for tablet ff7fc0d7664c430b97e75744e78185c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:29 UTC) from {username='oryx', principal='oryx/127.2.70.253@KRBTEST.COM'} at 127.0.0.1:36394
I20260925 21:56:29.973492 9791 raft_consensus.cc:2229] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:29.973544 9791 raft_consensus.cc:2258] T ff7fc0d7664c430b97e75744e78185c2 P 0882610850c9425ea77ae50c3a43dfbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:29.973682 9792 tablet_replica.cc:330] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba: stopping tablet replica
I20260925 21:56:29.973713 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 9174
I20260925 21:56:29.974239 9790 tablet_replica.cc:330] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d: stopping tablet replica
I20260925 21:56:29.974326 9790 raft_consensus.cc:2229] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:29.974367 9790 raft_consensus.cc:2258] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:29.974436 9792 raft_consensus.cc:2229] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:29.974524 9792 raft_consensus.cc:2258] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:29.974846 9790 ts_tablet_manager.cc:1905] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:29.975782 9792 ts_tablet_manager.cc:1905] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:29.976467 9790 ts_tablet_manager.cc:1918] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:29.976516 9790 log.cc:1196] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/ff7fc0d7664c430b97e75744e78185c2
I20260925 21:56:29.976733 9790 ts_tablet_manager.cc:1939] T ff7fc0d7664c430b97e75744e78185c2 P 0be7ac9f2c9d4849b2f19f16a277448d: Deleting consensus metadata
I20260925 21:56:29.977137 9011 catalog_manager.cc:4841] TS 0be7ac9f2c9d4849b2f19f16a277448d (127.2.70.194:35501): tablet ff7fc0d7664c430b97e75744e78185c2 (table test-table [id=f2a581e87c3b4e2e8a0794c09d415c31]) successfully deleted
I20260925 21:56:29.977376 9792 ts_tablet_manager.cc:1918] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:29.977432 9792 log.cc:1196] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/ff7fc0d7664c430b97e75744e78185c2
I20260925 21:56:29.977627 9792 ts_tablet_manager.cc:1939] T ff7fc0d7664c430b97e75744e78185c2 P d15c2de96cec44be9ed402dbaead6aba: Deleting consensus metadata
I20260925 21:56:29.978004 9014 catalog_manager.cc:4841] TS d15c2de96cec44be9ed402dbaead6aba (127.2.70.195:39491): tablet ff7fc0d7664c430b97e75744e78185c2 (table test-table [id=f2a581e87c3b4e2e8a0794c09d415c31]) successfully deleted
I20260925 21:56:29.981009 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 9401
I20260925 21:56:29.981526 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:29.981097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:43971 (local address 127.0.0.1:52658)
0925 21:56:29.981467 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:29.981494 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:43971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
W20260925 21:56:29.981776 9011 catalog_manager.cc:4584] TS 0882610850c9425ea77ae50c3a43dfbb (127.2.70.193:43971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff7fc0d7664c430b97e75744e78185c2: Network error: Client connection negotiation failed: client connection to 127.2.70.193:43971: connect: Connection refused (error 111)
I20260925 21:56:29.988426 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 9537
I20260925 21:56:29.993216 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 8893
I20260925 21:56:29.997454 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 9081
2026-09-25T21:56:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6176 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:30 dist-test-slave-2g0f krb5kdc[9795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:30 dist-test-slave-2g0f krb5kdc[9795](info): set up 2 sockets
Sep 25 21:56:30 dist-test-slave-2g0f krb5kdc[9795](info): commencing operation
krb5kdc: starting...
W20260925 21:56:30.964342 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.254.keytab.
Entry for principal oryx/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.254.keytab.
I20260925 21:56:31.083200 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:44193
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.254.keytab
--principal=oryx/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:42363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:44193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:31.166383 9808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:31.166605 9808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:31.166635 9808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:31.167871 9808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:31.167912 9808 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 21:56:31.167929 9808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:31.167945 9808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:31.167961 9808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:31.167977 9808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:31.169523 9808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:44193
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.254.keytab
--principal=oryx/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:44193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:31.169776 9808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:31.169993 9808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:31.174927 9814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:31.174953 9813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:31.174998 9816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:31.175318 9808 server_base.cc:1034] running on GCE node
I20260925 21:56:31.175561 9808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:31.175870 9808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:31.177095 9808 hybrid_clock.cc:648] HybridClock initialized: now 1790373391177079 us; error 25 us; skew 500 ppm
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373391, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:31.179530 9808 init.cc:377] Logged in from keytab as oryx/127.2.70.254@KRBTEST.COM (short username oryx)
I20260925 21:56:31.179991 9808 webserver.cc:460] Webserver started at http://127.2.70.254:39141/ using document root <none> and password file <none>
I20260925 21:56:31.180208 9808 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:31.180255 9808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:31.180382 9808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:31.181479 9808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "f0da2e59e9b04181880770f8879b38e8"
format_stamp: "Formatted at 2026-09-25 21:56:31 on dist-test-slave-2g0f"
server_key: "37dfaffe5ba7a7071b7ccd69bc2f1262"
server_key_iv: "cf95c1c4cd46bf43a4b3a06cb0318b8d"
server_key_version: "encryptionkey@0"
I20260925 21:56:31.181795 9808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "f0da2e59e9b04181880770f8879b38e8"
format_stamp: "Formatted at 2026-09-25 21:56:31 on dist-test-slave-2g0f"
server_key: "37dfaffe5ba7a7071b7ccd69bc2f1262"
server_key_iv: "cf95c1c4cd46bf43a4b3a06cb0318b8d"
server_key_version: "encryptionkey@0"
I20260925 21:56:31.183135 9808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 21:56:31.183811 9823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:31.184054 9808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:31.184149 9808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "f0da2e59e9b04181880770f8879b38e8"
format_stamp: "Formatted at 2026-09-25 21:56:31 on dist-test-slave-2g0f"
server_key: "37dfaffe5ba7a7071b7ccd69bc2f1262"
server_key_iv: "cf95c1c4cd46bf43a4b3a06cb0318b8d"
server_key_version: "encryptionkey@0"
I20260925 21:56:31.184214 9808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:31.200275 9808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:31.202451 9808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:31.202594 9808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:31.207063 9808 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:44193
I20260925 21:56:31.207139 9885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:44193 every 8 connection(s)
I20260925 21:56:31.207441 9808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:31.207854 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 9808
I20260925 21:56:31.207952 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:31.208153 2331 external_mini_cluster.cc:1428] Setting key 1df585d4718d8d2d3156e74396053848
I20260925 21:56:31.208199 9886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:31.211503 9886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8: Bootstrap starting.
I20260925 21:56:31.212316 9886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:31.212589 9886 log.cc:824] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:31.213277 9886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8: No bootstrap required, opened a new log
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
I20260925 21:56:31.215394 9886 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da2e59e9b04181880770f8879b38e8" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44193 } }
I20260925 21:56:31.215540 9886 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:31.215575 9886 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0da2e59e9b04181880770f8879b38e8, State: Initialized, Role: FOLLOWER
I20260925 21:56:31.215703 9886 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da2e59e9b04181880770f8879b38e8" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44193 } }
I20260925 21:56:31.215801 9886 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:31.215862 9886 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:31.215907 9886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:31.216511 9886 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da2e59e9b04181880770f8879b38e8" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44193 } }
I20260925 21:56:31.216625 9886 leader_election.cc:302] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0da2e59e9b04181880770f8879b38e8; no voters:
I20260925 21:56:31.216794 9886 leader_election.cc:288] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:31.216828 9891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:31.216938 9891 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [term 1 LEADER]: Becoming Leader. State: Replica: f0da2e59e9b04181880770f8879b38e8, State: Running, Role: LEADER
I20260925 21:56:31.217078 9886 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:31.217084 9891 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da2e59e9b04181880770f8879b38e8" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44193 } }
I20260925 21:56:31.217464 9891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0da2e59e9b04181880770f8879b38e8. Latest consensus state: current_term: 1 leader_uuid: "f0da2e59e9b04181880770f8879b38e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da2e59e9b04181880770f8879b38e8" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44193 } } }
I20260925 21:56:31.217540 9891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:31.217679 9893 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0da2e59e9b04181880770f8879b38e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da2e59e9b04181880770f8879b38e8" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 44193 } } }
I20260925 21:56:31.217736 9893 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:31.218328 9897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:31.218972 9897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:31.219287 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.208776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.2.70.254:44193)
0925 21:56:31.208949 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.208957 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.208984 (+ 27us) server_negotiation.cc:407] Connection header received
0925 21:56:31.209384 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.209391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.209438 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.209575 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:31.210331 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.211252 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.212283 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.212763 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.212869 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.215495 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:31.215499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:31.215499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:31.215510 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:31.217580 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.217999 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.218001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.218002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.218105 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.218316 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.218317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.218318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.218677 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:31.218837 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.218959 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.219099 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260925 21:56:31.220571 9897 catalog_manager.cc:1347] Generated new cluster ID: 369a5c3178a9427aa68f81ceedde19fb
I20260925 21:56:31.220634 9897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:31.262055 9897 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:31.262691 9897 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:31.270943 9897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8: Generated new TSK 0
I20260925 21:56:31.271152 9897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab.
Entry for principal oryx/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab.
I20260925 21:56:31.312052 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:44193
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab
--principal=oryx/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:42363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:31.393592 9912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:31.393795 9912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:31.393818 9912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:31.394973 9912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:31.395017 9912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:31.395037 9912 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 21:56:31.395053 9912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:31.396579 9912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab
--principal=oryx/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.254:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:31.396823 9912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:31.397050 9912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:31.401945 9918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:31.402072 9920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:31.401935 9917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:31.402293 9912 server_base.cc:1034] running on GCE node
I20260925 21:56:31.402521 9912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:31.402734 9912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:31.403924 9912 hybrid_clock.cc:648] HybridClock initialized: now 1790373391403908 us; error 28 us; skew 500 ppm
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373391, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:31.406438 9912 init.cc:377] Logged in from keytab as oryx/127.2.70.193@KRBTEST.COM (short username oryx)
I20260925 21:56:31.406868 9912 webserver.cc:460] Webserver started at http://127.2.70.193:43683/ using document root <none> and password file <none>
I20260925 21:56:31.407100 9912 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:31.407143 9912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:31.407238 9912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:31.408322 9912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "d7c8332309e74f249753eac08b0e5b49"
format_stamp: "Formatted at 2026-09-25 21:56:31 on dist-test-slave-2g0f"
server_key: "7c3312abba73cebe645b183dcbde3337"
server_key_iv: "642b8b38acb768c7a4e16f8dc965bae1"
server_key_version: "encryptionkey@0"
I20260925 21:56:31.408617 9912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "d7c8332309e74f249753eac08b0e5b49"
format_stamp: "Formatted at 2026-09-25 21:56:31 on dist-test-slave-2g0f"
server_key: "7c3312abba73cebe645b183dcbde3337"
server_key_iv: "642b8b38acb768c7a4e16f8dc965bae1"
server_key_version: "encryptionkey@0"
I20260925 21:56:31.409787 9912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:31.410511 9927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:31.410732 9912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:31.410812 9912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "d7c8332309e74f249753eac08b0e5b49"
format_stamp: "Formatted at 2026-09-25 21:56:31 on dist-test-slave-2g0f"
server_key: "7c3312abba73cebe645b183dcbde3337"
server_key_iv: "642b8b38acb768c7a4e16f8dc965bae1"
server_key_version: "encryptionkey@0"
I20260925 21:56:31.410867 9912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:31.425580 9912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:31.427656 9912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:31.427778 9912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:31.428333 9912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:31.428377 9912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:31.428400 9912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:31.428411 9912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373391, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
I20260925 21:56:31.435154 9912 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:40211
I20260925 21:56:31.435225 10042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:40211 every 8 connection(s)
I20260925 21:56:31.435552 9912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:31.437469 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 9912
I20260925 21:56:31.437575 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:31.437755 2331 external_mini_cluster.cc:1428] Setting key 561938819059e4944e713217e1f4191d
I20260925 21:56:31.437985 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.428795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:54021 (local address 127.2.70.254:44193)
0925 21:56:31.428858 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.428861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.429105 (+ 244us) server_negotiation.cc:407] Connection header received
0925 21:56:31.429605 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.429606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.429620 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.429694 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:31.430426 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.430974 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.432029 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.432235 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.432293 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.435179 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:31.435181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:31.435181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:31.435189 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:31.436639 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.437026 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.437027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.437028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.437085 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.437318 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.437319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.437320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.437440 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:31.437497 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.437798 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.437895 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 21:56:31.438138 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.428759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:44193 (local address 127.2.70.193:54021)
0925 21:56:31.428970 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.429012 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.429504 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.429779 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.429783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.430126 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:31.430322 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.430327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.431049 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.431050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.431950 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.431953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.432308 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.432309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.432446 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.432988 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:31.432999 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:31.435086 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:31.436733 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.436740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.436742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.436956 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.437143 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.437144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.437144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.437262 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.437497 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:31.437504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:31.437666 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.437763 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.437924 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":143,"threads_started":1}
I20260925 21:56:31.438580 10043 heartbeater.cc:347] Connected to a master server at 127.2.70.254:44193
I20260925 21:56:31.438663 10043 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:31.438835 10043 heartbeater.cc:511] Master 127.2.70.254:44193 requested a full tablet report, sending...
I20260925 21:56:31.439378 9839 ts_manager.cc:194] Registered new tserver with Master: d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211)
I20260925 21:56:31.440256 9839 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:54021
I20260925 21:56:31.445632 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.439914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:51139 (local address 127.2.70.254:44193)
0925 21:56:31.439951 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.439952 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.439961 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:31.439981 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.439982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.439992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.440054 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:31.440451 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.441091 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.442022 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.442206 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.442262 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.443252 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:31.443253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:31.443253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:31.443259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:31.444597 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.444960 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.444961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.444961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.445009 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.445189 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.445189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.445189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.445285 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:31.445332 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.445507 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.445567 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 21:56:31.445690 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.439649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:44193 (local address 127.2.70.193:51139)
0925 21:56:31.439772 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.439782 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.439854 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.440061 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.440062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.440254 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:31.440315 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.440355 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.441207 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.441208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.441931 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.441934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.442279 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.442280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.442345 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.442793 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:31.442798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:31.443163 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:31.444677 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.444678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.444679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.444887 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.445066 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.445067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.445067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.445133 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.445343 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:31.445344 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:31.445403 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.445474 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.445581 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab.
Entry for principal oryx/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab.
I20260925 21:56:31.466944 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:44193
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab
--principal=oryx/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:42363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:31.547792 10052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:31.548041 10052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:31.548065 10052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:31.549279 10052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:31.549326 10052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:31.549352 10052 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 21:56:31.549369 10052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:31.550954 10052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab
--principal=oryx/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.254:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:31.551165 10052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:31.551353 10052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:56:31.556416 10052 server_base.cc:1034] running on GCE node
W20260925 21:56:31.556350 10058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:31.556360 10057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:31.556331 10060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:31.556877 10052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:31.557174 10052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:31.558317 10052 hybrid_clock.cc:648] HybridClock initialized: now 1790373391558305 us; error 28 us; skew 500 ppm
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373391, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:31.560869 10052 init.cc:377] Logged in from keytab as oryx/127.2.70.194@KRBTEST.COM (short username oryx)
I20260925 21:56:31.561398 10052 webserver.cc:460] Webserver started at http://127.2.70.194:44221/ using document root <none> and password file <none>
I20260925 21:56:31.561619 10052 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:31.561671 10052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:31.561800 10052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:31.562850 10052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "6746f6a5ba7b413bbe4428842b5001fb"
format_stamp: "Formatted at 2026-09-25 21:56:31 on dist-test-slave-2g0f"
server_key: "0383b84ef2119545dccb012b4997b3c5"
server_key_iv: "01663a2061f3b4436cc2d3a8a29679f1"
server_key_version: "encryptionkey@0"
I20260925 21:56:31.563205 10052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "6746f6a5ba7b413bbe4428842b5001fb"
format_stamp: "Formatted at 2026-09-25 21:56:31 on dist-test-slave-2g0f"
server_key: "0383b84ef2119545dccb012b4997b3c5"
server_key_iv: "01663a2061f3b4436cc2d3a8a29679f1"
server_key_version: "encryptionkey@0"
I20260925 21:56:31.564566 10052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:31.565313 10067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:31.565479 10052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:31.565562 10052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "6746f6a5ba7b413bbe4428842b5001fb"
format_stamp: "Formatted at 2026-09-25 21:56:31 on dist-test-slave-2g0f"
server_key: "0383b84ef2119545dccb012b4997b3c5"
server_key_iv: "01663a2061f3b4436cc2d3a8a29679f1"
server_key_version: "encryptionkey@0"
I20260925 21:56:31.565629 10052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:31.574124 10052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:31.576160 10052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:31.576303 10052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:31.576778 10052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:31.576823 10052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:31.576856 10052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:31.576881 10052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:31.583475 10052 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:41313
I20260925 21:56:31.583597 10182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:41313 every 8 connection(s)
I20260925 21:56:31.583906 10052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373391, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
I20260925 21:56:31.587081 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.577342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:35433 (local address 127.2.70.254:44193)
0925 21:56:31.577406 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.577408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.577605 (+ 197us) server_negotiation.cc:407] Connection header received
0925 21:56:31.578188 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.578189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.578202 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.578273 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:31.579182 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.580023 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.581678 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.581877 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.581942 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.584619 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:31.584621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:31.584621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:31.584630 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:31.585768 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.586149 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.586150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.586151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.586213 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.586477 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.586479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.586479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.586633 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:31.586688 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.586937 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.587012 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 21:56:31.587234 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.577332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:44193 (local address 127.2.70.194:35433)
0925 21:56:31.577490 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.577526 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.578083 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.578387 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.578392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.578774 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:31.579089 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.579097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.580135 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.580136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.581593 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.581597 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.581966 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.581967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.582121 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.582764 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:31.582775 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:31.584527 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:31.585850 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.585856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.585859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.586077 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.586270 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.586271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.586271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.586388 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.586686 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:31.586691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:31.586830 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.586914 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.587040 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260925 21:56:31.587673 10183 heartbeater.cc:347] Connected to a master server at 127.2.70.254:44193
I20260925 21:56:31.587770 10183 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:31.587992 10183 heartbeater.cc:511] Master 127.2.70.254:44193 requested a full tablet report, sending...
I20260925 21:56:31.588405 9840 ts_manager.cc:194] Registered new tserver with Master: 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194:41313)
I20260925 21:56:31.588876 9840 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:35433
I20260925 21:56:31.591704 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 10052
I20260925 21:56:31.591850 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:31.592090 2331 external_mini_cluster.cc:1428] Setting key 29a99264d83bbf6ff6e12b0163bd99ef
I20260925 21:56:31.594080 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.588748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:51049 (local address 127.2.70.254:44193)
0925 21:56:31.588787 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.588789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.588909 (+ 120us) server_negotiation.cc:407] Connection header received
0925 21:56:31.588934 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.588935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.588944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.589000 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:31.589394 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.589881 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.590601 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.590799 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.590855 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.591692 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:31.591694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:31.591694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:31.591701 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:31.593050 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.593343 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.593344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.593344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.593397 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.593560 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.593561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.593562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.593711 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:31.593772 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.593922 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.593987 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 21:56:31.594086 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.588678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:44193 (local address 127.2.70.194:51049)
0925 21:56:31.588842 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.588852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.588901 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.589045 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.589046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.589252 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:31.589320 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.589322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.589983 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.589984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.590513 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.590515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.590858 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.590859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.590922 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.591300 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:31.591304 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:31.591613 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:31.593124 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.593125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.593125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.593279 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.593451 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.593452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.593452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.593509 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.593775 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:31.593776 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:31.593823 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.593885 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.593989 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for oryx/127.2.70.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab.
Entry for principal oryx/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab.
I20260925 21:56:31.616796 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:44193
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab
--principal=oryx/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:42363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:31.697271 10192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:31.697481 10192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:31.697513 10192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:31.698676 10192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:31.698716 10192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:31.698742 10192 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 21:56:31.698765 10192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:31.700412 10192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab
--principal=oryx/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.254:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:31.700664 10192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:31.700879 10192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:31.705960 10200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:31.705978 10197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:31.706110 10192 server_base.cc:1034] running on GCE node
W20260925 21:56:31.705978 10198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:31.706442 10192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:31.706679 10192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:31.707862 10192 hybrid_clock.cc:648] HybridClock initialized: now 1790373391707811 us; error 62 us; skew 500 ppm
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373391, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:31.710434 10192 init.cc:377] Logged in from keytab as oryx/127.2.70.195@KRBTEST.COM (short username oryx)
I20260925 21:56:31.710875 10192 webserver.cc:460] Webserver started at http://127.2.70.195:46421/ using document root <none> and password file <none>
I20260925 21:56:31.711146 10192 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:31.711211 10192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:31.711328 10192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:31.712368 10192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "01ea670a9a2745a4b6bf22de9f330a47"
format_stamp: "Formatted at 2026-09-25 21:56:31 on dist-test-slave-2g0f"
server_key: "2bd403e8a6e46c157ec70912200e4b5c"
server_key_iv: "6bfb423400de42c40189b5ff7e25ce0b"
server_key_version: "encryptionkey@0"
I20260925 21:56:31.712669 10192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "01ea670a9a2745a4b6bf22de9f330a47"
format_stamp: "Formatted at 2026-09-25 21:56:31 on dist-test-slave-2g0f"
server_key: "2bd403e8a6e46c157ec70912200e4b5c"
server_key_iv: "6bfb423400de42c40189b5ff7e25ce0b"
server_key_version: "encryptionkey@0"
I20260925 21:56:31.714030 10192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:31.714706 10207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:31.714875 10192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:31.715010 10192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "01ea670a9a2745a4b6bf22de9f330a47"
format_stamp: "Formatted at 2026-09-25 21:56:31 on dist-test-slave-2g0f"
server_key: "2bd403e8a6e46c157ec70912200e4b5c"
server_key_iv: "6bfb423400de42c40189b5ff7e25ce0b"
server_key_version: "encryptionkey@0"
I20260925 21:56:31.715075 10192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:31.726325 10192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:31.728402 10192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:31.728518 10192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:31.728981 10192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:31.729017 10192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:31.729036 10192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:31.729048 10192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:31.735034 10192 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:37191
I20260925 21:56:31.735172 10322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:37191 every 8 connection(s)
I20260925 21:56:31.735420 10192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373391, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
I20260925 21:56:31.738531 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.729437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:60857 (local address 127.2.70.254:44193)
0925 21:56:31.729514 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.729516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.729677 (+ 161us) server_negotiation.cc:407] Connection header received
0925 21:56:31.730155 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.730156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.730168 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.730247 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:31.731062 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.731818 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.733321 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.733632 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.733705 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.736415 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:31.736416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:31.736416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:31.736428 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:31.737479 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.737810 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.737810 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.737811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.737856 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.738016 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.738017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.738017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.738117 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:31.738165 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.738381 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.738444 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 21:56:31.738657 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.729385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:44193 (local address 127.2.70.195:60857)
0925 21:56:31.729549 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.729590 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.730065 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.730306 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.730310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.730677 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:31.730959 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.730967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.731930 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.731931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.733223 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.733227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.733709 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.733710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.733838 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.734498 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:31.734517 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:31.736311 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:31.737548 (+ 1237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.737553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.737555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.737753 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.737893 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.737894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.737894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.737978 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.738163 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:31.738166 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:31.738277 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.738352 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.738476 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260925 21:56:31.739142 10323 heartbeater.cc:347] Connected to a master server at 127.2.70.254:44193
I20260925 21:56:31.739225 10323 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:31.739437 10323 heartbeater.cc:511] Master 127.2.70.254:44193 requested a full tablet report, sending...
I20260925 21:56:31.739933 9839 ts_manager.cc:194] Registered new tserver with Master: 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195:37191)
I20260925 21:56:31.740411 9839 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:60857
I20260925 21:56:31.741689 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 10192
I20260925 21:56:31.741773 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:31.741971 2331 external_mini_cluster.cc:1428] Setting key 01fe29c28cce463f54ed23380a246176
I20260925 21:56:31.742853 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:56:31.746567 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.740157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:44193 (local address 127.2.70.195:54395)
0925 21:56:31.740271 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.740289 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.740348 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.740471 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.740472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.740666 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:31.740733 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.740735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.741563 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.741564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.742219 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.742222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.742641 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.742642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.742736 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.743156 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:31.743160 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:31.743550 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:31.745449 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.745450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.745451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.745618 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.745924 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.745925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.745925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.745989 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.746279 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:31.746281 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:31.746326 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.746392 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.746487 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":61,"threads_started":1}
I20260925 21:56:31.746589 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.740238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:54395 (local address 127.2.70.254:44193)
0925 21:56:31.740278 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.740280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.740335 (+ 55us) server_negotiation.cc:407] Connection header received
0925 21:56:31.740414 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.740414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.740422 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.740470 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:31.740810 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.741461 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.742296 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.742577 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.742625 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.743659 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:31.743660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:31.743660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:31.743667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:31.745321 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.745698 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.745699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.745699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.745763 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.746059 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.746059 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.746060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.746193 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:31.746260 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.746425 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.746497 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373391, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 21:56:31.764315 9889 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:31.758854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.2.70.254:44193)
0925 21:56:31.758930 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.758932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.758941 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:31.758970 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.758971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.758981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.759026 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:31.759577 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.760131 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.760876 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.761060 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.761118 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.764228 (+ 3110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58082: BlockingRecv error: recv got EOF from 127.0.0.1:58082 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373391, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
I20260925 21:56:31.772007 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.765069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.2.70.254:44193)
0925 21:56:31.765115 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.765117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.765126 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:31.765189 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.765190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.765205 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.765248 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:31.765711 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.766247 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.766954 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.767131 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.767187 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.769517 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:31.769519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:31.769519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:31.769526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:31.770794 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.771196 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.771197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.771197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.771248 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.771413 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.771414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.771414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.771573 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:31.771652 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.771862 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.771929 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 21:56:31.773742 9839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:31.775424 9839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:31.782027 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.777966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.2.70.195:37191)
0925 21:56:31.778130 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.778135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.778292 (+ 157us) server_negotiation.cc:407] Connection header received
0925 21:56:31.778362 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.778371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.778385 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.778464 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:31.778789 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.779919 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.781018 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.781578 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.781641 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.781718 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.781875 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.781920 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260925 21:56:31.782017 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.777769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:40211 (local address 127.0.0.1:60040)
0925 21:56:31.778373 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.778380 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.778421 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.778572 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.778573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.778575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:31.778644 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.778646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.779589 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.779590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.780887 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.780890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.781788 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.781789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.781841 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.781845 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.781882 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.781938 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":28,"thread_start_us":86,"threads_started":1}
I20260925 21:56:31.782042 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.778010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.2.70.193:40211)
0925 21:56:31.778172 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.778176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.778431 (+ 255us) server_negotiation.cc:407] Connection header received
0925 21:56:31.778484 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.778487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.778500 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.778562 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:31.778719 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.779507 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.780971 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.781708 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.781787 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.781858 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.781899 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.781941 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260925 21:56:31.782385 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.778082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.2.70.194:41313)
0925 21:56:31.778227 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.778230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.778241 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:56:31.778362 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.778364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.778381 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.778464 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:31.778738 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.779907 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.781143 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.781836 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.781895 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.781958 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.782271 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.782305 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260925 21:56:31.782217 10346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.777768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:41313 (local address 127.0.0.1:45860)
0925 21:56:31.778154 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.778196 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.778263 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.778542 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.778545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.778550 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:31.778657 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.778659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.779990 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.779990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.781068 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.781070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.781991 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.781993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.782053 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.782057 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.782107 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.782149 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":49,"thread_start_us":79,"threads_started":1}
I20260925 21:56:31.782284 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.777769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:37191 (local address 127.0.0.1:34938)
0925 21:56:31.778242 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.778251 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.778298 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.778542 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.778545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.778547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:31.778707 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.778708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.780004 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.780005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.780921 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.780924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.781660 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.781661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.781752 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.781768 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.782181 (+ 413us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.782229 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":84,"thread_start_us":85,"threads_started":1}
I20260925 21:56:31.783254 10257 tablet_service.cc:1467] Processing CreateTablet for tablet 263b43971e7d4ccdac5900aad3998cb4 (DEFAULT_TABLE table=test-table [id=88185a1dc0e74c04a956c9c98ee1297b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:31.783253 9977 tablet_service.cc:1467] Processing CreateTablet for tablet 263b43971e7d4ccdac5900aad3998cb4 (DEFAULT_TABLE table=test-table [id=88185a1dc0e74c04a956c9c98ee1297b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:31.783264 10117 tablet_service.cc:1467] Processing CreateTablet for tablet 263b43971e7d4ccdac5900aad3998cb4 (DEFAULT_TABLE table=test-table [id=88185a1dc0e74c04a956c9c98ee1297b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:31.783619 10117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263b43971e7d4ccdac5900aad3998cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:31.783622 9977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263b43971e7d4ccdac5900aad3998cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:31.783622 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263b43971e7d4ccdac5900aad3998cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:31.786777 10351 tablet_bootstrap.cc:492] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb: Bootstrap starting.
I20260925 21:56:31.787344 10352 tablet_bootstrap.cc:492] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47: Bootstrap starting.
I20260925 21:56:31.787768 10351 tablet_bootstrap.cc:654] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:31.787979 10353 tablet_bootstrap.cc:492] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49: Bootstrap starting.
I20260925 21:56:31.788094 10352 tablet_bootstrap.cc:654] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:31.788126 10351 log.cc:824] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:31.788408 10352 log.cc:824] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:31.788611 10353 tablet_bootstrap.cc:654] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:31.788913 10351 tablet_bootstrap.cc:492] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb: No bootstrap required, opened a new log
I20260925 21:56:31.788911 10353 log.cc:824] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:31.788985 10351 ts_tablet_manager.cc:1397] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:56:31.789096 10352 tablet_bootstrap.cc:492] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47: No bootstrap required, opened a new log
I20260925 21:56:31.789153 10352 ts_tablet_manager.cc:1397] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 21:56:31.789557 10353 tablet_bootstrap.cc:492] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49: No bootstrap required, opened a new log
I20260925 21:56:31.789623 10353 ts_tablet_manager.cc:1397] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:31.791009 10353 raft_consensus.cc:348] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.791147 10353 raft_consensus.cc:374] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:31.791179 10353 raft_consensus.cc:729] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c8332309e74f249753eac08b0e5b49, State: Initialized, Role: FOLLOWER
I20260925 21:56:31.791092 10352 raft_consensus.cc:348] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.791092 10351 raft_consensus.cc:348] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.791239 10352 raft_consensus.cc:374] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:31.791239 10351 raft_consensus.cc:374] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:31.791271 10352 raft_consensus.cc:729] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ea670a9a2745a4b6bf22de9f330a47, State: Initialized, Role: FOLLOWER
I20260925 21:56:31.791271 10351 raft_consensus.cc:729] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6746f6a5ba7b413bbe4428842b5001fb, State: Initialized, Role: FOLLOWER
I20260925 21:56:31.791316 10353 consensus_queue.cc:260] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.791404 10352 consensus_queue.cc:260] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.791389 10351 consensus_queue.cc:260] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.791638 10352 ts_tablet_manager.cc:1428] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 21:56:31.791638 10351 ts_tablet_manager.cc:1428] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:56:31.791626 10043 heartbeater.cc:503] Master 127.2.70.254:44193 was elected leader, sending a full tablet report...
I20260925 21:56:31.791747 10323 heartbeater.cc:503] Master 127.2.70.254:44193 was elected leader, sending a full tablet report...
I20260925 21:56:31.791759 10183 heartbeater.cc:503] Master 127.2.70.254:44193 was elected leader, sending a full tablet report...
I20260925 21:56:31.791620 10353 ts_tablet_manager.cc:1428] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260925 21:56:31.834582 10184 tablet.cc:2369] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:31.840951 10358 raft_consensus.cc:482] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:31.841099 10358 raft_consensus.cc:504] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.841449 10358 leader_election.cc:288] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211), 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194:41313)
I20260925 21:56:31.845420 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.841729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:54023 (local address 127.2.70.193:40211)
0925 21:56:31.841772 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.841774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.841812 (+ 38us) server_negotiation.cc:407] Connection header received
0925 21:56:31.841841 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.841842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.841851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.841901 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:31.842065 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.842831 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.844435 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.845004 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.845092 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.845099 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.845300 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.845347 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 21:56:31.845433 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.841714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:57031 (local address 127.2.70.194:41313)
0925 21:56:31.841753 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.841755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.841893 (+ 138us) server_negotiation.cc:407] Connection header received
0925 21:56:31.841922 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.841923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.841934 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.841988 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:31.842165 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.842846 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.844435 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.845004 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.845090 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.845099 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.845300 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.845377 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 21:56:31.845425 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.841645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:40211 (local address 127.2.70.195:54023)
0925 21:56:31.841699 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.841710 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.841754 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.841922 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.841923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.841925 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:31.841991 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.841992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.842946 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.842947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.844311 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.844315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.845120 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.845121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.845212 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.845218 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.845277 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.845329 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":91}
I20260925 21:56:31.845513 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.841663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:41313 (local address 127.2.70.195:57031)
0925 21:56:31.841823 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.841830 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.841873 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.842044 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.842045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.842047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:31.842099 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.842100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.842945 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.842947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.844313 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.844315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.845120 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.845121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.845212 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.845218 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.845277 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.845329 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":25,"thread_start_us":114,"threads_started":1}
I20260925 21:56:31.845826 10137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263b43971e7d4ccdac5900aad3998cb4" candidate_uuid: "01ea670a9a2745a4b6bf22de9f330a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6746f6a5ba7b413bbe4428842b5001fb" is_pre_election: true
I20260925 21:56:31.845820 9997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263b43971e7d4ccdac5900aad3998cb4" candidate_uuid: "01ea670a9a2745a4b6bf22de9f330a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c8332309e74f249753eac08b0e5b49" is_pre_election: true
I20260925 21:56:31.846001 10137 raft_consensus.cc:2454] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01ea670a9a2745a4b6bf22de9f330a47 in term 0.
I20260925 21:56:31.846035 9997 raft_consensus.cc:2454] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01ea670a9a2745a4b6bf22de9f330a47 in term 0.
I20260925 21:56:31.846256 10210 leader_election.cc:302] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ea670a9a2745a4b6bf22de9f330a47, d7c8332309e74f249753eac08b0e5b49; no voters:
I20260925 21:56:31.846443 10358 raft_consensus.cc:2781] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:31.846484 10358 raft_consensus.cc:482] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:31.846498 10358 raft_consensus.cc:3037] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:31.847388 10358 raft_consensus.cc:504] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.847517 10358 leader_election.cc:288] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [CANDIDATE]: Term 1 election: Requested vote from peers d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211), 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194:41313)
I20260925 21:56:31.847698 10137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263b43971e7d4ccdac5900aad3998cb4" candidate_uuid: "01ea670a9a2745a4b6bf22de9f330a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6746f6a5ba7b413bbe4428842b5001fb"
I20260925 21:56:31.847767 9997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263b43971e7d4ccdac5900aad3998cb4" candidate_uuid: "01ea670a9a2745a4b6bf22de9f330a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c8332309e74f249753eac08b0e5b49"
I20260925 21:56:31.847779 10137 raft_consensus.cc:3037] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:31.847878 9997 raft_consensus.cc:3037] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:31.848505 10137 raft_consensus.cc:2454] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01ea670a9a2745a4b6bf22de9f330a47 in term 1.
I20260925 21:56:31.848562 9997 raft_consensus.cc:2454] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01ea670a9a2745a4b6bf22de9f330a47 in term 1.
I20260925 21:56:31.848706 10209 leader_election.cc:302] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ea670a9a2745a4b6bf22de9f330a47, 6746f6a5ba7b413bbe4428842b5001fb; no voters:
I20260925 21:56:31.848802 10358 raft_consensus.cc:2781] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:31.848970 10358 raft_consensus.cc:686] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 LEADER]: Becoming Leader. State: Replica: 01ea670a9a2745a4b6bf22de9f330a47, State: Running, Role: LEADER
I20260925 21:56:31.849054 10358 consensus_queue.cc:237] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.849885 9838 catalog_manager.cc:5495] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195). New cstate: current_term: 1 leader_uuid: "01ea670a9a2745a4b6bf22de9f330a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } health_report { overall_health: HEALTHY } } }
I20260925 21:56:31.855047 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.852767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.2.70.195:37191)
0925 21:56:31.852804 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.852806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.852815 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:31.852876 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.852877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.852893 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.852946 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:31.853050 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.853496 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.854258 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.854441 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.854519 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.854695 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:31.854793 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:31.854871 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.854963 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.854982 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 21:56:31.856869 9997 raft_consensus.cc:1261] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Refusing update from remote peer 01ea670a9a2745a4b6bf22de9f330a47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:31.856918 10137 raft_consensus.cc:1261] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Refusing update from remote peer 01ea670a9a2745a4b6bf22de9f330a47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:31.857265 10361 consensus_queue.cc:1035] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:31.857400 10358 consensus_queue.cc:1035] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:31.859915 10366 mvcc.cc:205] Tried to move back new op lower bound from 7333369413042642944 to 7333369413014265856. Current Snapshot: MvccSnapshot[applied={T|T < 7333369413042642944}]
I20260925 21:56:31.860620 10365 mvcc.cc:205] Tried to move back new op lower bound from 7333369413042642944 to 7333369413014265856. Current Snapshot: MvccSnapshot[applied={T|T < 7333369413042642944}]
I20260925 21:56:31.862262 9838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:31.862344 9838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:31.863065 9838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8: Sending DeleteTablet for 3 replicas of tablet 263b43971e7d4ccdac5900aad3998cb4
I20260925 21:56:31.863373 10257 tablet_service.cc:1514] Processing DeleteTablet for tablet 263b43971e7d4ccdac5900aad3998cb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:31 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:34938
I20260925 21:56:31.863373 9977 tablet_service.cc:1514] Processing DeleteTablet for tablet 263b43971e7d4ccdac5900aad3998cb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:31 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:60040
I20260925 21:56:31.863502 10117 tablet_service.cc:1514] Processing DeleteTablet for tablet 263b43971e7d4ccdac5900aad3998cb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:31 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:45860
I20260925 21:56:31.863708 10376 tablet_replica.cc:330] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb: stopping tablet replica
I20260925 21:56:31.863797 10376 raft_consensus.cc:2229] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:31.863960 10376 raft_consensus.cc:2258] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:31.864030 9838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:31.864176 9838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:31.864297 10375 tablet_replica.cc:330] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49: stopping tablet replica
I20260925 21:56:31.863977 10374 tablet_replica.cc:330] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47: stopping tablet replica
I20260925 21:56:31.864500 10374 raft_consensus.cc:2229] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:31.864533 10376 ts_tablet_manager.cc:1905] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:31.864830 10375 raft_consensus.cc:2229] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:31.865027 10374 raft_consensus.cc:2258] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:31.865387 9977 tablet_service.cc:1467] Processing CreateTablet for tablet d5331ddf9ee54ae985bec8c4620c2cc9 (DEFAULT_TABLE table=test-table [id=c4572d8d8dc548209b75c8e8eb314dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:31.865418 10117 tablet_service.cc:1467] Processing CreateTablet for tablet d5331ddf9ee54ae985bec8c4620c2cc9 (DEFAULT_TABLE table=test-table [id=c4572d8d8dc548209b75c8e8eb314dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:31.865512 9977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5331ddf9ee54ae985bec8c4620c2cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:31.865523 10117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5331ddf9ee54ae985bec8c4620c2cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:31.866382 10353 tablet_bootstrap.cc:492] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49: Bootstrap starting.
I20260925 21:56:31.866730 10351 tablet_bootstrap.cc:492] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb: Bootstrap starting.
I20260925 21:56:31.866789 10353 tablet_bootstrap.cc:654] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:31.867129 10351 tablet_bootstrap.cc:654] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:31.867488 10376 ts_tablet_manager.cc:1918] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:31.867558 10376 log.cc:1196] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/263b43971e7d4ccdac5900aad3998cb4
I20260925 21:56:31.867663 10353 tablet_bootstrap.cc:492] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49: No bootstrap required, opened a new log
I20260925 21:56:31.867712 10353 ts_tablet_manager.cc:1397] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:31.867854 10376 ts_tablet_manager.cc:1939] T 263b43971e7d4ccdac5900aad3998cb4 P 6746f6a5ba7b413bbe4428842b5001fb: Deleting consensus metadata
I20260925 21:56:31.865415 10257 tablet_service.cc:1467] Processing CreateTablet for tablet d5331ddf9ee54ae985bec8c4620c2cc9 (DEFAULT_TABLE table=test-table [id=c4572d8d8dc548209b75c8e8eb314dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:31.868028 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5331ddf9ee54ae985bec8c4620c2cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:31.868139 10351 tablet_bootstrap.cc:492] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb: No bootstrap required, opened a new log
I20260925 21:56:31.868173 10351 ts_tablet_manager.cc:1397] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:31.868275 9825 catalog_manager.cc:4841] TS 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194:41313): tablet 263b43971e7d4ccdac5900aad3998cb4 (table test-table [id=88185a1dc0e74c04a956c9c98ee1297b]) successfully deleted
I20260925 21:56:31.868306 10351 raft_consensus.cc:348] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.868366 10351 raft_consensus.cc:374] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:31.868383 10351 raft_consensus.cc:729] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6746f6a5ba7b413bbe4428842b5001fb, State: Initialized, Role: FOLLOWER
I20260925 21:56:31.868423 10351 consensus_queue.cc:260] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.868517 10351 ts_tablet_manager.cc:1428] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:31.869287 10352 tablet_bootstrap.cc:492] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47: Bootstrap starting.
I20260925 21:56:31.869673 10352 tablet_bootstrap.cc:654] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:31.869740 10375 raft_consensus.cc:2258] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:31.869702 10353 raft_consensus.cc:348] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.869814 10353 raft_consensus.cc:374] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:31.869840 10353 raft_consensus.cc:729] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c8332309e74f249753eac08b0e5b49, State: Initialized, Role: FOLLOWER
I20260925 21:56:31.869886 10353 consensus_queue.cc:260] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.870036 10353 ts_tablet_manager.cc:1428] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:31.870204 10374 ts_tablet_manager.cc:1905] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:31.871953 10375 ts_tablet_manager.cc:1905] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:31.873674 10375 ts_tablet_manager.cc:1918] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:31.873755 10375 log.cc:1196] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/wal/wals/263b43971e7d4ccdac5900aad3998cb4
I20260925 21:56:31.874004 10375 ts_tablet_manager.cc:1939] T 263b43971e7d4ccdac5900aad3998cb4 P d7c8332309e74f249753eac08b0e5b49: Deleting consensus metadata
I20260925 21:56:31.874432 10374 ts_tablet_manager.cc:1918] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:31.874461 9826 catalog_manager.cc:4841] TS d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211): tablet 263b43971e7d4ccdac5900aad3998cb4 (table test-table [id=88185a1dc0e74c04a956c9c98ee1297b]) successfully deleted
I20260925 21:56:31.874506 10374 log.cc:1196] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/263b43971e7d4ccdac5900aad3998cb4
I20260925 21:56:31.874734 10374 ts_tablet_manager.cc:1939] T 263b43971e7d4ccdac5900aad3998cb4 P 01ea670a9a2745a4b6bf22de9f330a47: Deleting consensus metadata
I20260925 21:56:31.875149 9824 catalog_manager.cc:4841] TS 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195:37191): tablet 263b43971e7d4ccdac5900aad3998cb4 (table test-table [id=88185a1dc0e74c04a956c9c98ee1297b]) successfully deleted
I20260925 21:56:31.876237 10352 tablet_bootstrap.cc:492] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47: No bootstrap required, opened a new log
I20260925 21:56:31.876282 10352 ts_tablet_manager.cc:1397] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260925 21:56:31.876415 10352 raft_consensus.cc:348] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.876483 10352 raft_consensus.cc:374] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:31.876495 10352 raft_consensus.cc:729] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ea670a9a2745a4b6bf22de9f330a47, State: Initialized, Role: FOLLOWER
I20260925 21:56:31.876533 10352 consensus_queue.cc:260] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.876618 10352 ts_tablet_manager.cc:1428] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260925 21:56:31.936488 10044 tablet.cc:2369] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:31.937284 10359 raft_consensus.cc:482] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:31.937361 10359 raft_consensus.cc:504] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.937696 10359 leader_election.cc:288] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211), 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195:37191)
I20260925 21:56:31.941193 10380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.937892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:37191 (local address 127.2.70.194:45097)
0925 21:56:31.938008 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.938017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.938087 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.938298 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.938298 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.938300 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:31.938367 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.938369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.939101 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.939102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.940379 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.940380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.940938 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.940940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.941003 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.941007 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.941052 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.941094 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260925 21:56:31.941183 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.938069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:45097 (local address 127.2.70.195:37191)
0925 21:56:31.938114 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.938116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.938126 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:31.938183 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.938184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.938193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.938254 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:31.938423 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.939010 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.940469 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.940837 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.940899 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.940909 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.941078 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.941115 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 21:56:31.941385 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.937880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:40211 (local address 127.2.70.194:50979)
0925 21:56:31.937945 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.937956 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.938004 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.938244 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.938245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.938246 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:31.938313 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.938315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.939213 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.939214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.940360 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.940364 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.941129 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.941130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.941219 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.941223 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.941271 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.941313 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":3}
I20260925 21:56:31.941596 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.938109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:50979 (local address 127.2.70.193:40211)
0925 21:56:31.938150 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.938151 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.938158 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:56:31.938183 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.938184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.938193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.938240 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:31.938391 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.939136 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.940455 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.941010 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.941084 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.941093 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.941489 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.941534 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 21:56:31.941718 10277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5331ddf9ee54ae985bec8c4620c2cc9" candidate_uuid: "6746f6a5ba7b413bbe4428842b5001fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ea670a9a2745a4b6bf22de9f330a47" is_pre_election: true
I20260925 21:56:31.941794 9997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5331ddf9ee54ae985bec8c4620c2cc9" candidate_uuid: "6746f6a5ba7b413bbe4428842b5001fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c8332309e74f249753eac08b0e5b49" is_pre_election: true
I20260925 21:56:31.941849 10277 raft_consensus.cc:2454] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6746f6a5ba7b413bbe4428842b5001fb in term 0.
I20260925 21:56:31.941879 9997 raft_consensus.cc:2454] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6746f6a5ba7b413bbe4428842b5001fb in term 0.
I20260925 21:56:31.942062 10068 leader_election.cc:302] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ea670a9a2745a4b6bf22de9f330a47, 6746f6a5ba7b413bbe4428842b5001fb; no voters:
I20260925 21:56:31.942206 10359 raft_consensus.cc:2781] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:31.942253 10359 raft_consensus.cc:482] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:31.942272 10359 raft_consensus.cc:3037] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:31.942950 10359 raft_consensus.cc:504] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.943079 10359 leader_election.cc:288] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [CANDIDATE]: Term 1 election: Requested vote from peers d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211), 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195:37191)
I20260925 21:56:31.943267 9997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5331ddf9ee54ae985bec8c4620c2cc9" candidate_uuid: "6746f6a5ba7b413bbe4428842b5001fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c8332309e74f249753eac08b0e5b49"
I20260925 21:56:31.943316 10277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5331ddf9ee54ae985bec8c4620c2cc9" candidate_uuid: "6746f6a5ba7b413bbe4428842b5001fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ea670a9a2745a4b6bf22de9f330a47"
I20260925 21:56:31.943356 9997 raft_consensus.cc:3037] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:31.943384 10277 raft_consensus.cc:3037] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:31.944057 10277 raft_consensus.cc:2454] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6746f6a5ba7b413bbe4428842b5001fb in term 1.
I20260925 21:56:31.944191 10068 leader_election.cc:302] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ea670a9a2745a4b6bf22de9f330a47, 6746f6a5ba7b413bbe4428842b5001fb; no voters:
I20260925 21:56:31.944200 9997 raft_consensus.cc:2454] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6746f6a5ba7b413bbe4428842b5001fb in term 1.
I20260925 21:56:31.944267 10359 raft_consensus.cc:2781] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:31.944399 10359 raft_consensus.cc:686] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 LEADER]: Becoming Leader. State: Replica: 6746f6a5ba7b413bbe4428842b5001fb, State: Running, Role: LEADER
I20260925 21:56:31.944483 10359 consensus_queue.cc:237] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.945248 9838 catalog_manager.cc:5495] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194). New cstate: current_term: 1 leader_uuid: "6746f6a5ba7b413bbe4428842b5001fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } health_report { overall_health: UNKNOWN } } }
Sep 25 21:56:31 dist-test-slave-2g0f krb5kdc[9795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373391, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.254@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
W20260925 21:56:31.985919 10324 tablet.cc:2369] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:31.987370 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.980591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:44193 (local address 127.0.0.1:58112)
0925 21:56:31.980725 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:31.980736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:31.980795 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:31.980915 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:31.980916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:31.981190 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:31.981241 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.981243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.981852 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.981853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.982405 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:31.982407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.982704 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.982705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:31.982742 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.983247 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:31.983252 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:31.984621 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:31.986212 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.986219 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.986220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.986408 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.986593 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:31.986594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:31.986594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:31.986676 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:31.987016 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:31.987022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:31.987072 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 21:56:31.987154 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 21:56:31.987287 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260925 21:56:31.987459 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:31.980685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.2.70.254:44193)
0925 21:56:31.980763 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:31.980765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:31.980778 (+ 13us) server_negotiation.cc:407] Connection header received
0925 21:56:31.980851 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:31.980852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:31.980860 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:31.980899 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:31.981303 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.981783 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.982474 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:31.982651 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:31.982697 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:31.984709 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:31.984711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:31.984711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:31.984717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:31.986122 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.986469 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.986470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.986470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.986518 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:31.986730 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:31.986730 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:31.986730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:31.986845 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:31.986975 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:31.987195 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 21:56:31.987355 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":58}
I20260925 21:56:31.989531 9838 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:58112:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 21:56:31.989727 9838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:31.991148 10117 tablet_service.cc:1467] Processing CreateTablet for tablet 0dfe427dd4dc45e88b63a3a73e3856b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe99cf65800e4f3e8d17b61d04f85524]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 21:56:31.991148 10257 tablet_service.cc:1467] Processing CreateTablet for tablet 0dfe427dd4dc45e88b63a3a73e3856b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe99cf65800e4f3e8d17b61d04f85524]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 21:56:31.991142 9977 tablet_service.cc:1467] Processing CreateTablet for tablet 0dfe427dd4dc45e88b63a3a73e3856b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe99cf65800e4f3e8d17b61d04f85524]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 21:56:31.991290 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dfe427dd4dc45e88b63a3a73e3856b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:31.991290 9977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dfe427dd4dc45e88b63a3a73e3856b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:31.991297 10117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dfe427dd4dc45e88b63a3a73e3856b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:31.992554 10351 tablet_bootstrap.cc:492] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb: Bootstrap starting.
I20260925 21:56:31.992753 10353 tablet_bootstrap.cc:492] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49: Bootstrap starting.
I20260925 21:56:31.992934 10351 tablet_bootstrap.cc:654] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:31.993108 10353 tablet_bootstrap.cc:654] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:31.993420 10352 tablet_bootstrap.cc:492] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: Bootstrap starting.
I20260925 21:56:31.993608 10351 tablet_bootstrap.cc:492] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb: No bootstrap required, opened a new log
I20260925 21:56:31.993656 10351 ts_tablet_manager.cc:1397] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:31.993721 10352 tablet_bootstrap.cc:654] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:31.993801 10351 raft_consensus.cc:348] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.993877 10351 raft_consensus.cc:374] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:31.993903 10351 raft_consensus.cc:729] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6746f6a5ba7b413bbe4428842b5001fb, State: Initialized, Role: FOLLOWER
I20260925 21:56:31.993979 10351 consensus_queue.cc:260] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.994082 10351 ts_tablet_manager.cc:1428] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:31.994239 10352 tablet_bootstrap.cc:492] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: No bootstrap required, opened a new log
I20260925 21:56:31.994206 10359 tablet_replica.cc:439] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } } }
I20260925 21:56:31.994273 10359 tablet_replica.cc:442] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:31.994282 10352 ts_tablet_manager.cc:1397] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:31.994397 10352 raft_consensus.cc:348] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.994457 10352 raft_consensus.cc:374] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:31.994474 10352 raft_consensus.cc:729] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ea670a9a2745a4b6bf22de9f330a47, State: Initialized, Role: FOLLOWER
I20260925 21:56:31.994515 10352 consensus_queue.cc:260] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.994596 10352 ts_tablet_manager.cc:1428] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:31.994598 10353 tablet_bootstrap.cc:492] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49: No bootstrap required, opened a new log
I20260925 21:56:31.994642 10353 ts_tablet_manager.cc:1397] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:31.994637 10371 tablet_replica.cc:439] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } } }
I20260925 21:56:31.994728 10371 tablet_replica.cc:442] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:31.994776 10353 raft_consensus.cc:348] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.994861 10353 raft_consensus.cc:374] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:31.994886 10353 raft_consensus.cc:729] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c8332309e74f249753eac08b0e5b49, State: Initialized, Role: FOLLOWER
I20260925 21:56:31.994928 10353 consensus_queue.cc:260] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:31.995011 10353 ts_tablet_manager.cc:1428] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:31.995067 10357 tablet_replica.cc:439] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } } }
I20260925 21:56:31.995158 10357 tablet_replica.cc:442] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:32.032454 10357 raft_consensus.cc:482] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:32.032550 10357 raft_consensus.cc:504] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.032872 10357 leader_election.cc:288] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194:41313), 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195:37191)
I20260925 21:56:32.034955 10371 raft_consensus.cc:482] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:32.035040 10371 raft_consensus.cc:504] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.035188 10371 leader_election.cc:288] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211), 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194:41313)
I20260925 21:56:32.035346 9997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfe427dd4dc45e88b63a3a73e3856b4" candidate_uuid: "01ea670a9a2745a4b6bf22de9f330a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c8332309e74f249753eac08b0e5b49" is_pre_election: true
I20260925 21:56:32.035373 10137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfe427dd4dc45e88b63a3a73e3856b4" candidate_uuid: "01ea670a9a2745a4b6bf22de9f330a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6746f6a5ba7b413bbe4428842b5001fb" is_pre_election: true
I20260925 21:56:32.035426 9997 raft_consensus.cc:2454] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01ea670a9a2745a4b6bf22de9f330a47 in term 0.
I20260925 21:56:32.035445 10137 raft_consensus.cc:2454] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01ea670a9a2745a4b6bf22de9f330a47 in term 0.
I20260925 21:56:32.035564 10210 leader_election.cc:302] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ea670a9a2745a4b6bf22de9f330a47, d7c8332309e74f249753eac08b0e5b49; no voters:
I20260925 21:56:32.035632 10371 raft_consensus.cc:2781] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:32.035662 10371 raft_consensus.cc:482] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:32.035681 10371 raft_consensus.cc:3037] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:32.036661 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.033181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:37949 (local address 127.2.70.195:37191)
0925 21:56:32.033235 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:32.033237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:32.033351 (+ 114us) server_negotiation.cc:407] Connection header received
0925 21:56:32.033403 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:32.033405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:32.033418 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:32.033478 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:32.033636 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.034371 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.035910 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.036317 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.036369 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.036376 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:32.036549 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 21:56:32.036595 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 21:56:32.036737 10371 raft_consensus.cc:504] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.036796 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.033162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:59843 (local address 127.2.70.194:41313)
0925 21:56:32.033241 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:32.033243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:32.033333 (+ 90us) server_negotiation.cc:407] Connection header received
0925 21:56:32.033404 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:32.033404 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:32.033414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:32.033504 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:32.033628 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.034371 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.035980 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.036463 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.036537 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.036547 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:32.036687 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 21:56:32.036738 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 21:56:32.036798 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.033097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:41313 (local address 127.2.70.193:59843)
0925 21:56:32.033247 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:32.033257 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:32.033334 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:32.033507 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:32.033508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:32.033509 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:32.033562 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:32.033565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.034465 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.034466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:32.035785 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:32.035786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.036551 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.036552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:32.036616 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.036619 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:32.036666 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 21:56:32.036711 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":66,"thread_start_us":80,"threads_started":1}
I20260925 21:56:32.036859 10371 leader_election.cc:288] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [CANDIDATE]: Term 1 election: Requested vote from peers d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211), 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194:41313)
I20260925 21:56:32.036911 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.033107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:37191 (local address 127.2.70.193:37949)
0925 21:56:32.033280 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:32.033289 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:32.033340 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:32.033489 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:32.033490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:32.033492 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:32.033576 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:32.033577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.034465 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.034466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:32.035782 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:32.035786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.036425 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.036426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:32.036483 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.036486 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:32.036742 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 21:56:32.036775 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":9,"thread_start_us":67,"threads_started":1}
I20260925 21:56:32.037063 10137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfe427dd4dc45e88b63a3a73e3856b4" candidate_uuid: "01ea670a9a2745a4b6bf22de9f330a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6746f6a5ba7b413bbe4428842b5001fb"
I20260925 21:56:32.037127 10137 raft_consensus.cc:3037] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:32.037214 9997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfe427dd4dc45e88b63a3a73e3856b4" candidate_uuid: "01ea670a9a2745a4b6bf22de9f330a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c8332309e74f249753eac08b0e5b49"
I20260925 21:56:32.037281 9997 raft_consensus.cc:3037] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:32.037744 10137 raft_consensus.cc:2454] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01ea670a9a2745a4b6bf22de9f330a47 in term 1.
I20260925 21:56:32.037858 9997 raft_consensus.cc:2454] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01ea670a9a2745a4b6bf22de9f330a47 in term 1.
I20260925 21:56:32.037842 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfe427dd4dc45e88b63a3a73e3856b4" candidate_uuid: "d7c8332309e74f249753eac08b0e5b49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6746f6a5ba7b413bbe4428842b5001fb" is_pre_election: true
I20260925 21:56:32.037925 10136 raft_consensus.cc:2379] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c8332309e74f249753eac08b0e5b49 in current term 1: Already voted for candidate 01ea670a9a2745a4b6bf22de9f330a47 in this term.
I20260925 21:56:32.037942 10209 leader_election.cc:302] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ea670a9a2745a4b6bf22de9f330a47, 6746f6a5ba7b413bbe4428842b5001fb; no voters:
I20260925 21:56:32.038079 10371 raft_consensus.cc:2781] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:32.038125 10371 raft_consensus.cc:686] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 LEADER]: Becoming Leader. State: Replica: 01ea670a9a2745a4b6bf22de9f330a47, State: Running, Role: LEADER
I20260925 21:56:32.038143 10277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfe427dd4dc45e88b63a3a73e3856b4" candidate_uuid: "d7c8332309e74f249753eac08b0e5b49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ea670a9a2745a4b6bf22de9f330a47" is_pre_election: true
I20260925 21:56:32.038164 10371 consensus_queue.cc:237] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.038492 10361 tablet_replica.cc:439] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: TxnStatusTablet state changed. Reason: New leader 01ea670a9a2745a4b6bf22de9f330a47. Latest consensus state: current_term: 1 leader_uuid: "01ea670a9a2745a4b6bf22de9f330a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } } }
I20260925 21:56:32.038486 9928 leader_election.cc:302] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c8332309e74f249753eac08b0e5b49; no voters: 01ea670a9a2745a4b6bf22de9f330a47, 6746f6a5ba7b413bbe4428842b5001fb
I20260925 21:56:32.038566 10361 tablet_replica.cc:442] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: This TxnStatusTablet replica's current role is: LEADER
I20260925 21:56:32.038679 10357 raft_consensus.cc:2726] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 21:56:32.038798 10395 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 21:56:32.038913 10395 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 21:56:32.038980 9840 catalog_manager.cc:5495] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195). New cstate: current_term: 1 leader_uuid: "01ea670a9a2745a4b6bf22de9f330a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } health_report { overall_health: HEALTHY } } }
I20260925 21:56:32.055661 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.053433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:37191 (local address 127.0.0.1:34950)
0925 21:56:32.053511 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:32.053522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:32.053570 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:32.053695 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:32.053696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:32.053697 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 21:56:32.053749 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:32.053752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.054350 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.054351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:32.054927 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:32.054929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.055260 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.055261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:32.055324 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.055339 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 21:56:32.055500 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 21:56:32.055508 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 21:56:32.055549 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 21:56:32.055594 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 21:56:32.055655 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.053508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34950 (local address 127.2.70.195:37191)
0925 21:56:32.053578 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:32.053580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:32.053590 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:32.053626 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:32.053627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:32.053639 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:32.053685 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:32.053810 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.054277 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.055011 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.055197 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.055252 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.055420 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:32.055449 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:32.055489 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:32.055552 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 21:56:32.055571 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 21:56:32.056702 10136 raft_consensus.cc:1261] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Refusing update from remote peer 01ea670a9a2745a4b6bf22de9f330a47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:32.056764 9997 raft_consensus.cc:1261] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Refusing update from remote peer 01ea670a9a2745a4b6bf22de9f330a47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:32.056981 10361 consensus_queue.cc:1035] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:32.057108 10361 consensus_queue.cc:1035] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:32.057613 10357 tablet_replica.cc:439] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49: TxnStatusTablet state changed. Reason: New leader 01ea670a9a2745a4b6bf22de9f330a47. Latest consensus state: current_term: 1 leader_uuid: "01ea670a9a2745a4b6bf22de9f330a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } } }
I20260925 21:56:32.057703 10357 tablet_replica.cc:442] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:32.057654 10359 tablet_replica.cc:439] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb: TxnStatusTablet state changed. Reason: New leader 01ea670a9a2745a4b6bf22de9f330a47. Latest consensus state: current_term: 1 leader_uuid: "01ea670a9a2745a4b6bf22de9f330a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } } }
I20260925 21:56:32.057735 10359 tablet_replica.cc:442] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:32.058152 10371 tablet_replica.cc:439] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01ea670a9a2745a4b6bf22de9f330a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } } }
I20260925 21:56:32.058223 10371 tablet_replica.cc:442] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: This TxnStatusTablet replica's current role is: LEADER
I20260925 21:56:32.058529 10364 mvcc.cc:205] Tried to move back new op lower bound from 7333369413862428672 to 7333369413788635136. Current Snapshot: MvccSnapshot[applied={T|T < 7333369413862428672 or (T in {7333369413862428672})}]
I20260925 21:56:32.058920 10357 tablet_replica.cc:439] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01ea670a9a2745a4b6bf22de9f330a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } } }
I20260925 21:56:32.058995 10357 tablet_replica.cc:442] T 0dfe427dd4dc45e88b63a3a73e3856b4 P d7c8332309e74f249753eac08b0e5b49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:32.059283 10361 tablet_replica.cc:439] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01ea670a9a2745a4b6bf22de9f330a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } } }
I20260925 21:56:32.059351 10361 tablet_replica.cc:442] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 01ea670a9a2745a4b6bf22de9f330a47: This TxnStatusTablet replica's current role is: LEADER
I20260925 21:56:32.059988 10359 tablet_replica.cc:439] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01ea670a9a2745a4b6bf22de9f330a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } } }
I20260925 21:56:32.060070 10359 tablet_replica.cc:442] T 0dfe427dd4dc45e88b63a3a73e3856b4 P 6746f6a5ba7b413bbe4428842b5001fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 21:56:32.062503 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.059731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.2.70.194:41313)
0925 21:56:32.059773 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:32.059775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:32.059796 (+ 21us) server_negotiation.cc:407] Connection header received
0925 21:56:32.060078 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:32.060078 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:32.060097 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:32.060159 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:32.060347 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.060861 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.061627 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.061924 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.061996 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.062173 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:32.062311 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:32.062384 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:32.062405 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 21:56:32.062432 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 21:56:32.067267 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.065041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:37191 (local address 127.2.70.194:57907)
0925 21:56:32.065113 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:32.065124 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:32.065164 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:32.065320 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:32.065321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:32.065322 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 21:56:32.065388 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:32.065389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.065996 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.065997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:32.066574 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:32.066576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.066897 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.066898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:32.066962 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.066977 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 21:56:32.067124 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 21:56:32.067126 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:32.067165 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 21:56:32.067204 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 21:56:32.067265 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.065112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:57907 (local address 127.2.70.195:37191)
0925 21:56:32.065155 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:32.065158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:32.065166 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:32.065221 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:32.065222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:32.065232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:32.065277 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:32.065465 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.065924 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.066650 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.066837 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.066889 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.067045 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:32.067076 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:32.067114 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:32.067167 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 21:56:32.067188 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 21:56:32.069566 9997 raft_consensus.cc:1261] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Refusing update from remote peer 6746f6a5ba7b413bbe4428842b5001fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:32.069638 10277 raft_consensus.cc:1261] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Refusing update from remote peer 6746f6a5ba7b413bbe4428842b5001fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:32.069787 10359 consensus_queue.cc:1035] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:32.069919 10381 consensus_queue.cc:1035] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:32.078042 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.075592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:32987 (local address 127.2.70.194:41313)
0925 21:56:32.075629 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:32.075631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:32.075661 (+ 30us) server_negotiation.cc:407] Connection header received
0925 21:56:32.075706 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:32.075707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:32.075716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:32.075776 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:32.075893 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.076540 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.077345 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.077600 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.077656 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.077809 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:32.077844 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:32.077887 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:32.077944 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 21:56:32.077964 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 21:56:32.078053 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.075546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:41313 (local address 127.2.70.195:32987)
0925 21:56:32.075621 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:32.075631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:32.075668 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:32.075778 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:32.075779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:32.075780 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 21:56:32.075821 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:32.075823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.076610 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.076611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:32.077277 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:32.077279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.077667 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.077668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:32.077723 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.077741 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 21:56:32.077895 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 21:56:32.077897 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:32.077933 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 21:56:32.077988 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 21:56:32.087396 9840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:32.087530 9840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:32.088222 9840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8: Sending DeleteTablet for 3 replicas of tablet d5331ddf9ee54ae985bec8c4620c2cc9
I20260925 21:56:32.088466 10117 tablet_service.cc:1514] Processing DeleteTablet for tablet d5331ddf9ee54ae985bec8c4620c2cc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:32 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:45860
I20260925 21:56:32.088526 9977 tablet_service.cc:1514] Processing DeleteTablet for tablet d5331ddf9ee54ae985bec8c4620c2cc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:32 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:60040
I20260925 21:56:32.088619 10376 tablet_replica.cc:330] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb: stopping tablet replica
I20260925 21:56:32.088626 10375 tablet_replica.cc:330] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49: stopping tablet replica
I20260925 21:56:32.088663 10376 raft_consensus.cc:2229] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:32.088693 10375 raft_consensus.cc:2229] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:32.088732 10375 raft_consensus.cc:2258] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:32.088738 10376 raft_consensus.cc:2258] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:32.089020 10375 ts_tablet_manager.cc:1905] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:32.089095 10376 ts_tablet_manager.cc:1905] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:32.089221 10257 tablet_service.cc:1514] Processing DeleteTablet for tablet d5331ddf9ee54ae985bec8c4620c2cc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:32 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:34938
I20260925 21:56:32.090672 10376 ts_tablet_manager.cc:1918] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:32.090719 10376 log.cc:1196] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/d5331ddf9ee54ae985bec8c4620c2cc9
I20260925 21:56:32.090806 10375 ts_tablet_manager.cc:1918] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:32.090857 10375 log.cc:1196] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/wal/wals/d5331ddf9ee54ae985bec8c4620c2cc9
I20260925 21:56:32.090902 10376 ts_tablet_manager.cc:1939] T d5331ddf9ee54ae985bec8c4620c2cc9 P 6746f6a5ba7b413bbe4428842b5001fb: Deleting consensus metadata
I20260925 21:56:32.091066 10375 ts_tablet_manager.cc:1939] T d5331ddf9ee54ae985bec8c4620c2cc9 P d7c8332309e74f249753eac08b0e5b49: Deleting consensus metadata
I20260925 21:56:32.091198 10374 tablet_replica.cc:330] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47: stopping tablet replica
I20260925 21:56:32.091259 10374 raft_consensus.cc:2229] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:32.091296 10374 raft_consensus.cc:2258] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:32.091409 9825 catalog_manager.cc:4841] TS 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194:41313): tablet d5331ddf9ee54ae985bec8c4620c2cc9 (table test-table [id=c4572d8d8dc548209b75c8e8eb314dc1]) successfully deleted
I20260925 21:56:32.091682 10374 ts_tablet_manager.cc:1905] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:32.092515 9826 catalog_manager.cc:4841] TS d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211): tablet d5331ddf9ee54ae985bec8c4620c2cc9 (table test-table [id=c4572d8d8dc548209b75c8e8eb314dc1]) successfully deleted
I20260925 21:56:32.093317 10374 ts_tablet_manager.cc:1918] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:32.093369 10374 log.cc:1196] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/d5331ddf9ee54ae985bec8c4620c2cc9
I20260925 21:56:32.093570 10374 ts_tablet_manager.cc:1939] T d5331ddf9ee54ae985bec8c4620c2cc9 P 01ea670a9a2745a4b6bf22de9f330a47: Deleting consensus metadata
I20260925 21:56:32.093874 9824 catalog_manager.cc:4841] TS 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195:37191): tablet d5331ddf9ee54ae985bec8c4620c2cc9 (table test-table [id=c4572d8d8dc548209b75c8e8eb314dc1]) successfully deleted
Sep 25 21:56:32 dist-test-slave-2g0f krb5kdc[9795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 21:56:32 dist-test-slave-2g0f krb5kdc[9795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:32 dist-test-slave-2g0f krb5kdc[9795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 21:56:32.114995 9889 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:32.109753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58126 (local address 127.2.70.254:44193)
0925 21:56:32.109806 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:32.109808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:32.109927 (+ 119us) server_negotiation.cc:407] Connection header received
0925 21:56:32.109988 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:32.109989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:32.110000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:32.110046 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:32.110600 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.111100 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.111871 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.112061 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.112122 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.114928 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58126: BlockingRecv error: recv got EOF from 127.0.0.1:58126 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 21:56:32 dist-test-slave-2g0f krb5kdc[9795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM
I20260925 21:56:32.127367 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.115955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.2.70.254:44193)
0925 21:56:32.116001 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:32.116003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:32.120156 (+ 4153us) server_negotiation.cc:407] Connection header received
0925 21:56:32.120220 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:32.120221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:32.120233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:32.120283 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:32.120783 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.121274 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.122026 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.122216 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.122272 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.124456 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:32.124457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:32.124458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:32.124466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:32.126343 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:32.126653 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:32.126654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:32.126654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:32.126705 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:32.126873 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:32.126874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:32.126875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:32.126985 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:32.127071 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:32.127188 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 21:56:32.127273 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 21:56:32.129112 9840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:32.129273 9840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:32.130640 9977 tablet_service.cc:1467] Processing CreateTablet for tablet cb0de37f2b0c4868be4494312ab53c45 (DEFAULT_TABLE table=test-table [id=a19054af95d74c75841a2472566fd9a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:32.130769 9977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0de37f2b0c4868be4494312ab53c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:32.131713 10353 tablet_bootstrap.cc:492] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49: Bootstrap starting.
I20260925 21:56:32.131748 10257 tablet_service.cc:1467] Processing CreateTablet for tablet cb0de37f2b0c4868be4494312ab53c45 (DEFAULT_TABLE table=test-table [id=a19054af95d74c75841a2472566fd9a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:32.131886 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0de37f2b0c4868be4494312ab53c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:32.132382 10353 tablet_bootstrap.cc:654] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:32.132949 10353 tablet_bootstrap.cc:492] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49: No bootstrap required, opened a new log
I20260925 21:56:32.132990 10353 ts_tablet_manager.cc:1397] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:32.133139 10353 raft_consensus.cc:348] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.133208 10353 raft_consensus.cc:374] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:32.133225 10353 raft_consensus.cc:729] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c8332309e74f249753eac08b0e5b49, State: Initialized, Role: FOLLOWER
I20260925 21:56:32.133213 10117 tablet_service.cc:1467] Processing CreateTablet for tablet cb0de37f2b0c4868be4494312ab53c45 (DEFAULT_TABLE table=test-table [id=a19054af95d74c75841a2472566fd9a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:32.133265 10353 consensus_queue.cc:260] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.133318 10117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0de37f2b0c4868be4494312ab53c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:32.133343 10353 ts_tablet_manager.cc:1428] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:32.134265 10351 tablet_bootstrap.cc:492] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb: Bootstrap starting.
I20260925 21:56:32.134266 10352 tablet_bootstrap.cc:492] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47: Bootstrap starting.
I20260925 21:56:32.134559 10352 tablet_bootstrap.cc:654] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:32.134644 10351 tablet_bootstrap.cc:654] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:32.135627 10352 tablet_bootstrap.cc:492] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47: No bootstrap required, opened a new log
I20260925 21:56:32.135675 10352 ts_tablet_manager.cc:1397] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:32.135807 10352 raft_consensus.cc:348] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.135957 10352 raft_consensus.cc:374] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:32.135979 10352 raft_consensus.cc:729] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ea670a9a2745a4b6bf22de9f330a47, State: Initialized, Role: FOLLOWER
I20260925 21:56:32.136019 10352 consensus_queue.cc:260] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.136099 10352 ts_tablet_manager.cc:1428] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:32.136271 10351 tablet_bootstrap.cc:492] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb: No bootstrap required, opened a new log
I20260925 21:56:32.136310 10351 ts_tablet_manager.cc:1397] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:32.136456 10351 raft_consensus.cc:348] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.136526 10351 raft_consensus.cc:374] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:32.136543 10351 raft_consensus.cc:729] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6746f6a5ba7b413bbe4428842b5001fb, State: Initialized, Role: FOLLOWER
I20260925 21:56:32.136575 10351 consensus_queue.cc:260] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.136646 10351 ts_tablet_manager.cc:1428] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:32.440798 10405 raft_consensus.cc:482] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:32.440918 10405 raft_consensus.cc:504] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.441118 10405 leader_election.cc:288] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211), 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195:37191)
I20260925 21:56:32.441346 9997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0de37f2b0c4868be4494312ab53c45" candidate_uuid: "6746f6a5ba7b413bbe4428842b5001fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c8332309e74f249753eac08b0e5b49" is_pre_election: true
I20260925 21:56:32.441454 9997 raft_consensus.cc:2454] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6746f6a5ba7b413bbe4428842b5001fb in term 0.
I20260925 21:56:32.441385 10277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0de37f2b0c4868be4494312ab53c45" candidate_uuid: "6746f6a5ba7b413bbe4428842b5001fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ea670a9a2745a4b6bf22de9f330a47" is_pre_election: true
I20260925 21:56:32.441500 10277 raft_consensus.cc:2454] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6746f6a5ba7b413bbe4428842b5001fb in term 0.
I20260925 21:56:32.441666 10070 leader_election.cc:302] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6746f6a5ba7b413bbe4428842b5001fb, d7c8332309e74f249753eac08b0e5b49; no voters:
I20260925 21:56:32.441781 10405 raft_consensus.cc:2781] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:32.441816 10405 raft_consensus.cc:482] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:32.441834 10405 raft_consensus.cc:3037] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:32.442539 10405 raft_consensus.cc:504] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.442651 10405 leader_election.cc:288] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [CANDIDATE]: Term 1 election: Requested vote from peers d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211), 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195:37191)
I20260925 21:56:32.442831 9997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0de37f2b0c4868be4494312ab53c45" candidate_uuid: "6746f6a5ba7b413bbe4428842b5001fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c8332309e74f249753eac08b0e5b49"
I20260925 21:56:32.442853 10277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0de37f2b0c4868be4494312ab53c45" candidate_uuid: "6746f6a5ba7b413bbe4428842b5001fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ea670a9a2745a4b6bf22de9f330a47"
I20260925 21:56:32.442931 10277 raft_consensus.cc:3037] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:32.442931 9997 raft_consensus.cc:3037] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:32.443719 10277 raft_consensus.cc:2454] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6746f6a5ba7b413bbe4428842b5001fb in term 1.
I20260925 21:56:32.443713 9997 raft_consensus.cc:2454] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6746f6a5ba7b413bbe4428842b5001fb in term 1.
I20260925 21:56:32.443954 10070 leader_election.cc:302] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6746f6a5ba7b413bbe4428842b5001fb, d7c8332309e74f249753eac08b0e5b49; no voters:
I20260925 21:56:32.444131 10405 raft_consensus.cc:2781] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:32.444181 10405 raft_consensus.cc:686] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 LEADER]: Becoming Leader. State: Replica: 6746f6a5ba7b413bbe4428842b5001fb, State: Running, Role: LEADER
I20260925 21:56:32.444231 10405 consensus_queue.cc:237] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.444818 9840 catalog_manager.cc:5495] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194). New cstate: current_term: 1 leader_uuid: "6746f6a5ba7b413bbe4428842b5001fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } health_report { overall_health: UNKNOWN } } }
I20260925 21:56:32.492247 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.490067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.2.70.194:41313)
0925 21:56:32.490122 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:32.490125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:32.490134 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:32.490178 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:32.490179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:32.490190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:32.490238 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:32.490346 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.490841 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.491600 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.491780 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.491857 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.492023 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:32.492053 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:32.492094 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:32.492148 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 21:56:32.492170 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 21:56:32.493057 9997 raft_consensus.cc:1261] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Refusing update from remote peer 6746f6a5ba7b413bbe4428842b5001fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:32.493106 10277 raft_consensus.cc:1261] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Refusing update from remote peer 6746f6a5ba7b413bbe4428842b5001fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:32.493278 10359 consensus_queue.cc:1035] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:32.493409 10405 consensus_queue.cc:1035] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:32.495882 9838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:32.495973 9838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:32.496508 9838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8: Sending DeleteTablet for 3 replicas of tablet cb0de37f2b0c4868be4494312ab53c45
I20260925 21:56:32.496743 10117 tablet_service.cc:1514] Processing DeleteTablet for tablet cb0de37f2b0c4868be4494312ab53c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:32 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:45860
I20260925 21:56:32.496778 9977 tablet_service.cc:1514] Processing DeleteTablet for tablet cb0de37f2b0c4868be4494312ab53c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:32 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:60040
I20260925 21:56:32.496793 10257 tablet_service.cc:1514] Processing DeleteTablet for tablet cb0de37f2b0c4868be4494312ab53c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:32 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:34938
I20260925 21:56:32.496865 10375 tablet_replica.cc:330] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49: stopping tablet replica
I20260925 21:56:32.496873 10376 tablet_replica.cc:330] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb: stopping tablet replica
I20260925 21:56:32.496908 10375 raft_consensus.cc:2229] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:32.496922 10376 raft_consensus.cc:2229] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:32.496949 10375 raft_consensus.cc:2258] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:32.496982 10376 raft_consensus.cc:2258] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:32.497044 10374 tablet_replica.cc:330] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47: stopping tablet replica
I20260925 21:56:32.497093 10374 raft_consensus.cc:2229] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:32.497125 10374 raft_consensus.cc:2258] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:32.497110 9838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 21:56:32.497210 10375 ts_tablet_manager.cc:1905] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 21:56:32.497246 9838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:32.497359 10376 ts_tablet_manager.cc:1905] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:32.498095 10374 ts_tablet_manager.cc:1905] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:32.498767 10375 ts_tablet_manager.cc:1918] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:32.498767 10376 ts_tablet_manager.cc:1918] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:32.498821 10376 log.cc:1196] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/cb0de37f2b0c4868be4494312ab53c45
I20260925 21:56:32.498821 10375 log.cc:1196] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-0/wal/wals/cb0de37f2b0c4868be4494312ab53c45
I20260925 21:56:32.499030 10375 ts_tablet_manager.cc:1939] T cb0de37f2b0c4868be4494312ab53c45 P d7c8332309e74f249753eac08b0e5b49: Deleting consensus metadata
I20260925 21:56:32.499030 10376 ts_tablet_manager.cc:1939] T cb0de37f2b0c4868be4494312ab53c45 P 6746f6a5ba7b413bbe4428842b5001fb: Deleting consensus metadata
I20260925 21:56:32.499276 10374 ts_tablet_manager.cc:1918] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:32.499327 10374 log.cc:1196] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/cb0de37f2b0c4868be4494312ab53c45
I20260925 21:56:32.499353 9826 catalog_manager.cc:4841] TS d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211): tablet cb0de37f2b0c4868be4494312ab53c45 (table test-table [id=a19054af95d74c75841a2472566fd9a4]) successfully deleted
I20260925 21:56:32.499406 9825 catalog_manager.cc:4841] TS 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194:41313): tablet cb0de37f2b0c4868be4494312ab53c45 (table test-table [id=a19054af95d74c75841a2472566fd9a4]) successfully deleted
I20260925 21:56:32.499495 10374 ts_tablet_manager.cc:1939] T cb0de37f2b0c4868be4494312ab53c45 P 01ea670a9a2745a4b6bf22de9f330a47: Deleting consensus metadata
I20260925 21:56:32.499935 9824 catalog_manager.cc:4841] TS 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195:37191): tablet cb0de37f2b0c4868be4494312ab53c45 (table test-table [id=a19054af95d74c75841a2472566fd9a4]) successfully deleted
I20260925 21:56:32.500387 10257 tablet_service.cc:1467] Processing CreateTablet for tablet aed5581d46514a9d9c421036d642e5ed (DEFAULT_TABLE table=test-table [id=1dfe233cfd4c48be8df214b59f5dec22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:32.500387 10117 tablet_service.cc:1467] Processing CreateTablet for tablet aed5581d46514a9d9c421036d642e5ed (DEFAULT_TABLE table=test-table [id=1dfe233cfd4c48be8df214b59f5dec22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:32.500392 9977 tablet_service.cc:1467] Processing CreateTablet for tablet aed5581d46514a9d9c421036d642e5ed (DEFAULT_TABLE table=test-table [id=1dfe233cfd4c48be8df214b59f5dec22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:32.500484 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed5581d46514a9d9c421036d642e5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:32.500487 10117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed5581d46514a9d9c421036d642e5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:32.500511 9977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed5581d46514a9d9c421036d642e5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:32.501243 10353 tablet_bootstrap.cc:492] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49: Bootstrap starting.
I20260925 21:56:32.501537 10353 tablet_bootstrap.cc:654] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:32.501549 10352 tablet_bootstrap.cc:492] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47: Bootstrap starting.
I20260925 21:56:32.501547 10351 tablet_bootstrap.cc:492] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb: Bootstrap starting.
I20260925 21:56:32.501785 10351 tablet_bootstrap.cc:654] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:32.501861 10352 tablet_bootstrap.cc:654] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:32.502302 10351 tablet_bootstrap.cc:492] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb: No bootstrap required, opened a new log
I20260925 21:56:32.502341 10351 ts_tablet_manager.cc:1397] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 21:56:32.502432 10353 tablet_bootstrap.cc:492] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49: No bootstrap required, opened a new log
I20260925 21:56:32.502437 10351 raft_consensus.cc:348] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.502472 10353 ts_tablet_manager.cc:1397] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:32.502493 10351 raft_consensus.cc:374] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:32.502522 10351 raft_consensus.cc:729] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6746f6a5ba7b413bbe4428842b5001fb, State: Initialized, Role: FOLLOWER
I20260925 21:56:32.502569 10351 consensus_queue.cc:260] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.502609 10353 raft_consensus.cc:348] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.502660 10351 ts_tablet_manager.cc:1428] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:32.502672 10353 raft_consensus.cc:374] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:32.502691 10353 raft_consensus.cc:729] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c8332309e74f249753eac08b0e5b49, State: Initialized, Role: FOLLOWER
I20260925 21:56:32.502723 10353 consensus_queue.cc:260] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.502815 10353 ts_tablet_manager.cc:1428] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:32.502947 10352 tablet_bootstrap.cc:492] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47: No bootstrap required, opened a new log
I20260925 21:56:32.502986 10352 ts_tablet_manager.cc:1397] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 21:56:32.503077 10352 raft_consensus.cc:348] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.503142 10352 raft_consensus.cc:374] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:32.503166 10352 raft_consensus.cc:729] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ea670a9a2745a4b6bf22de9f330a47, State: Initialized, Role: FOLLOWER
I20260925 21:56:32.503207 10352 consensus_queue.cc:260] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.503293 10352 ts_tablet_manager.cc:1428] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:32.508455 10371 raft_consensus.cc:482] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:32.508524 10371 raft_consensus.cc:504] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.508625 10371 leader_election.cc:288] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211), 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194:41313)
I20260925 21:56:32.508766 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed5581d46514a9d9c421036d642e5ed" candidate_uuid: "01ea670a9a2745a4b6bf22de9f330a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6746f6a5ba7b413bbe4428842b5001fb" is_pre_election: true
I20260925 21:56:32.508837 10136 raft_consensus.cc:2454] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01ea670a9a2745a4b6bf22de9f330a47 in term 0.
I20260925 21:56:32.508940 9997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed5581d46514a9d9c421036d642e5ed" candidate_uuid: "01ea670a9a2745a4b6bf22de9f330a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c8332309e74f249753eac08b0e5b49" is_pre_election: true
I20260925 21:56:32.508958 10209 leader_election.cc:302] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ea670a9a2745a4b6bf22de9f330a47, 6746f6a5ba7b413bbe4428842b5001fb; no voters:
I20260925 21:56:32.509034 9997 raft_consensus.cc:2454] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01ea670a9a2745a4b6bf22de9f330a47 in term 0.
I20260925 21:56:32.509064 10371 raft_consensus.cc:2781] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:32.509111 10371 raft_consensus.cc:482] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:32.509131 10371 raft_consensus.cc:3037] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:32.509711 10371 raft_consensus.cc:504] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.509809 10371 leader_election.cc:288] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [CANDIDATE]: Term 1 election: Requested vote from peers d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211), 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194:41313)
I20260925 21:56:32.509933 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed5581d46514a9d9c421036d642e5ed" candidate_uuid: "01ea670a9a2745a4b6bf22de9f330a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6746f6a5ba7b413bbe4428842b5001fb"
I20260925 21:56:32.509997 10136 raft_consensus.cc:3037] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:32.510540 10136 raft_consensus.cc:2454] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01ea670a9a2745a4b6bf22de9f330a47 in term 1.
I20260925 21:56:32.510645 9997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed5581d46514a9d9c421036d642e5ed" candidate_uuid: "01ea670a9a2745a4b6bf22de9f330a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c8332309e74f249753eac08b0e5b49"
I20260925 21:56:32.510692 10209 leader_election.cc:302] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ea670a9a2745a4b6bf22de9f330a47, 6746f6a5ba7b413bbe4428842b5001fb; no voters:
I20260925 21:56:32.510718 9997 raft_consensus.cc:3037] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:32.510759 10371 raft_consensus.cc:2781] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:32.510800 10371 raft_consensus.cc:686] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 LEADER]: Becoming Leader. State: Replica: 01ea670a9a2745a4b6bf22de9f330a47, State: Running, Role: LEADER
I20260925 21:56:32.510843 10371 consensus_queue.cc:237] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } }
I20260925 21:56:32.511381 9997 raft_consensus.cc:2454] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01ea670a9a2745a4b6bf22de9f330a47 in term 1.
I20260925 21:56:32.511318 9840 catalog_manager.cc:5495] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195). New cstate: current_term: 1 leader_uuid: "01ea670a9a2745a4b6bf22de9f330a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01ea670a9a2745a4b6bf22de9f330a47" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 37191 } health_report { overall_health: HEALTHY } } }
I20260925 21:56:32.517627 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.515501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.2.70.195:37191)
0925 21:56:32.515549 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:32.515551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:32.515560 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:32.515597 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:32.515598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:32.515607 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:32.515638 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:32.515723 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.516233 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.517036 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.517244 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.517291 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.517443 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:32.517469 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:32.517500 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:32.517545 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 21:56:32.517559 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 21:56:32.521464 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.519546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:41995 (local address 127.2.70.195:37191)
0925 21:56:32.519570 (+ 24us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:32.519572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:32.519576 (+ 4us) server_negotiation.cc:407] Connection header received
0925 21:56:32.519587 (+ 11us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:32.519587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:32.519593 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:32.519623 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:32.519707 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.520241 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.520949 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.521147 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.521200 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.521289 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:32.521311 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:32.521340 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:32.521381 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 21:56:32.521393 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 21:56:32.521508 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.519417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:37191 (local address 127.2.70.195:41995)
0925 21:56:32.519460 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:32.519470 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:32.519503 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:32.519632 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:32.519633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:32.519634 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 21:56:32.519665 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:32.519666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.520302 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.520302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:32.520822 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:32.520824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:32.521205 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:32.521205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:32.521249 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:32.521252 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 21:56:32.521349 (+ 97us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 21:56:32.521350 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 21:56:32.521378 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 21:56:32.521411 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 21:56:32.523322 10136 raft_consensus.cc:1261] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Refusing update from remote peer 01ea670a9a2745a4b6bf22de9f330a47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:32.523325 9997 raft_consensus.cc:1261] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Refusing update from remote peer 01ea670a9a2745a4b6bf22de9f330a47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:32.523495 10371 consensus_queue.cc:1035] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c8332309e74f249753eac08b0e5b49" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:32.523593 10358 consensus_queue.cc:1035] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6746f6a5ba7b413bbe4428842b5001fb" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:32.532922 9838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:32.533027 9838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:32.533592 9838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0da2e59e9b04181880770f8879b38e8: Sending DeleteTablet for 3 replicas of tablet aed5581d46514a9d9c421036d642e5ed
I20260925 21:56:32.533829 10117 tablet_service.cc:1514] Processing DeleteTablet for tablet aed5581d46514a9d9c421036d642e5ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:32 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:45860
I20260925 21:56:32.533838 9977 tablet_service.cc:1514] Processing DeleteTablet for tablet aed5581d46514a9d9c421036d642e5ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:32 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:60040
I20260925 21:56:32.533937 10375 tablet_replica.cc:330] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49: stopping tablet replica
I20260925 21:56:32.533989 10375 raft_consensus.cc:2229] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:32.533987 10376 tablet_replica.cc:330] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb: stopping tablet replica
I20260925 21:56:32.534022 10375 raft_consensus.cc:2258] T aed5581d46514a9d9c421036d642e5ed P d7c8332309e74f249753eac08b0e5b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:32.534041 10376 raft_consensus.cc:2229] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:32.534083 10376 raft_consensus.cc:2258] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:32.534166 10256 tablet_service.cc:1514] Processing DeleteTablet for tablet aed5581d46514a9d9c421036d642e5ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:32 UTC) from {username='oryx', principal='oryx/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:34938
I20260925 21:56:32.534251 10374 tablet_replica.cc:330] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47: stopping tablet replica
I20260925 21:56:32.534302 10374 raft_consensus.cc:2229] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:32.534368 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 9912
I20260925 21:56:32.534365 10374 raft_consensus.cc:2258] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:32.535254 10374 ts_tablet_manager.cc:1905] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:32.536443 10376 ts_tablet_manager.cc:1905] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:32.536677 10374 ts_tablet_manager.cc:1918] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:32.536725 10374 log.cc:1196] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/aed5581d46514a9d9c421036d642e5ed
I20260925 21:56:32.536890 10374 ts_tablet_manager.cc:1939] T aed5581d46514a9d9c421036d642e5ed P 01ea670a9a2745a4b6bf22de9f330a47: Deleting consensus metadata
I20260925 21:56:32.537302 9824 catalog_manager.cc:4841] TS 01ea670a9a2745a4b6bf22de9f330a47 (127.2.70.195:37191): tablet aed5581d46514a9d9c421036d642e5ed (table test-table [id=1dfe233cfd4c48be8df214b59f5dec22]) successfully deleted
I20260925 21:56:32.537827 10376 ts_tablet_manager.cc:1918] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 21:56:32.537874 10376 log.cc:1196] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/aed5581d46514a9d9c421036d642e5ed
I20260925 21:56:32.538046 10376 ts_tablet_manager.cc:1939] T aed5581d46514a9d9c421036d642e5ed P 6746f6a5ba7b413bbe4428842b5001fb: Deleting consensus metadata
I20260925 21:56:32.538847 9825 catalog_manager.cc:4841] TS 6746f6a5ba7b413bbe4428842b5001fb (127.2.70.194:41313): tablet aed5581d46514a9d9c421036d642e5ed (table test-table [id=1dfe233cfd4c48be8df214b59f5dec22]) successfully deleted
W20260925 21:56:32.540921 9826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:40211 (error 108)
I20260925 21:56:32.541281 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 10052
I20260925 21:56:32.541416 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:32.541248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:40211 (local address 127.0.0.1:60056)
0925 21:56:32.541336 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:32.541377 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:40211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":40,"threads_started":1}
W20260925 21:56:32.542018 9826 catalog_manager.cc:4584] TS d7c8332309e74f249753eac08b0e5b49 (127.2.70.193:40211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aed5581d46514a9d9c421036d642e5ed: Network error: Client connection negotiation failed: client connection to 127.2.70.193:40211: connect: Connection refused (error 111)
I20260925 21:56:32.547240 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 10192
I20260925 21:56:32.552528 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 9808
2026-09-25T21:56:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2554 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:32 dist-test-slave-2g0f krb5kdc[10437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:32 dist-test-slave-2g0f krb5kdc[10437](info): set up 2 sockets
Sep 25 21:56:32 dist-test-slave-2g0f krb5kdc[10437](info): commencing operation
krb5kdc: starting...
W20260925 21:56:33.512130 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:33 dist-test-slave-2g0f krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:56:33.630935 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:41011
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:41011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:33.709952 10450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:33.710161 10450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:33.710181 10450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:33.711370 10450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:33.711414 10450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:33.711427 10450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:33.711438 10450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:33.711448 10450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:33.713016 10450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:41011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:41011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:33.713251 10450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:33.713470 10450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:33.718183 10458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:33.718222 10455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:33.718238 10456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:33.718351 10450 server_base.cc:1034] running on GCE node
I20260925 21:56:33.718640 10450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:33.718937 10450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:33.720100 10450 hybrid_clock.cc:648] HybridClock initialized: now 1790373393720080 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T21:56:33Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1176 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:33 dist-test-slave-2g0f krb5kdc[10463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:33 dist-test-slave-2g0f krb5kdc[10463](info): set up 2 sockets
Sep 25 21:56:33 dist-test-slave-2g0f krb5kdc[10463](info): commencing operation
krb5kdc: starting...
W20260925 21:56:34.704216 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:34 dist-test-slave-2g0f krb5kdc[10463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:56:34.819061 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:40965
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:40965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:34.898756 10476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:34.898988 10476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:34.899020 10476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:34.900229 10476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:34.900272 10476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:34.900291 10476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:34.900306 10476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:34.900321 10476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:34.901836 10476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:40965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:40965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:34.902086 10476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:34.902295 10476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:34.907029 10482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:34.907048 10484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:34.907158 10481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:34.907305 10476 server_base.cc:1034] running on GCE node
I20260925 21:56:34.907569 10476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:34.907891 10476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:34.909056 10476 hybrid_clock.cc:648] HybridClock initialized: now 1790373394909043 us; error 31 us; skew 500 ppm
Sep 25 21:56:34 dist-test-slave-2g0f krb5kdc[10463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373394, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:34.911526 10476 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:56:34.912056 10476 webserver.cc:460] Webserver started at http://127.2.70.254:38615/ using document root <none> and password file <none>
I20260925 21:56:34.912263 10476 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:34.912312 10476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:34.912457 10476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:34.913528 10476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "dca47c537e884fec9e18a8e1a212d4b2"
format_stamp: "Formatted at 2026-09-25 21:56:34 on dist-test-slave-2g0f"
server_key: "fc2738273692a89a2e9fc8928b26af77"
server_key_iv: "88254206e9404a0392527e0d5d266f39"
server_key_version: "encryptionkey@0"
I20260925 21:56:34.913861 10476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "dca47c537e884fec9e18a8e1a212d4b2"
format_stamp: "Formatted at 2026-09-25 21:56:34 on dist-test-slave-2g0f"
server_key: "fc2738273692a89a2e9fc8928b26af77"
server_key_iv: "88254206e9404a0392527e0d5d266f39"
server_key_version: "encryptionkey@0"
I20260925 21:56:34.915086 10476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 21:56:34.915771 10491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:34.915925 10476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:34.916018 10476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "dca47c537e884fec9e18a8e1a212d4b2"
format_stamp: "Formatted at 2026-09-25 21:56:34 on dist-test-slave-2g0f"
server_key: "fc2738273692a89a2e9fc8928b26af77"
server_key_iv: "88254206e9404a0392527e0d5d266f39"
server_key_version: "encryptionkey@0"
I20260925 21:56:34.916091 10476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:34.926177 10476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:34.928277 10476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:34.928417 10476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:34.932205 10476 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:40965
I20260925 21:56:34.932247 10543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:40965 every 8 connection(s)
I20260925 21:56:34.932578 10476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:34.933399 10544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:34.933561 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 10476
I20260925 21:56:34.933656 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:34.933827 2331 external_mini_cluster.cc:1428] Setting key d60d120d1cb882b004b5e2b8a10c855d
I20260925 21:56:34.936250 10544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: Bootstrap starting.
I20260925 21:56:34.936928 10544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:34.937212 10544 log.cc:824] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:34.937842 10544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: No bootstrap required, opened a new log
I20260925 21:56:34.939733 10544 raft_consensus.cc:348] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca47c537e884fec9e18a8e1a212d4b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40965 } }
I20260925 21:56:34.939909 10544 raft_consensus.cc:374] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:34.940054 10544 raft_consensus.cc:729] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca47c537e884fec9e18a8e1a212d4b2, State: Initialized, Role: FOLLOWER
I20260925 21:56:34.940194 10544 consensus_queue.cc:260] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca47c537e884fec9e18a8e1a212d4b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40965 } }
I20260925 21:56:34.940274 10544 raft_consensus.cc:388] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:34.940312 10544 raft_consensus.cc:482] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:34.940358 10544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 21:56:34 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:34.941072 10544 raft_consensus.cc:504] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca47c537e884fec9e18a8e1a212d4b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40965 } }
I20260925 21:56:34.941171 10544 leader_election.cc:302] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dca47c537e884fec9e18a8e1a212d4b2; no voters:
I20260925 21:56:34.941325 10544 leader_election.cc:288] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:34.941558 10544 sys_catalog.cc:564] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:34.941704 10549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:34.941869 10549 raft_consensus.cc:686] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 1 LEADER]: Becoming Leader. State: Replica: dca47c537e884fec9e18a8e1a212d4b2, State: Running, Role: LEADER
I20260925 21:56:34.942739 10549 consensus_queue.cc:237] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca47c537e884fec9e18a8e1a212d4b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40965 } }
I20260925 21:56:34.943289 10550 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dca47c537e884fec9e18a8e1a212d4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca47c537e884fec9e18a8e1a212d4b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40965 } } }
I20260925 21:56:34.943892 10550 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:34.943459 10551 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dca47c537e884fec9e18a8e1a212d4b2. Latest consensus state: current_term: 1 leader_uuid: "dca47c537e884fec9e18a8e1a212d4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca47c537e884fec9e18a8e1a212d4b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40965 } } }
I20260925 21:56:34.944101 10551 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [sys.catalog]: This master's current role is: LEADER
W20260925 21:56:34.943688 10563 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:56:34.944195 10563 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:56:34.944388 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:34.934396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.2.70.254:40965)
0925 21:56:34.934581 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:34.934590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:34.934630 (+ 40us) server_negotiation.cc:407] Connection header received
0925 21:56:34.935130 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:34.935139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:34.935203 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:34.935347 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:34.935928 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:34.937223 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:34.938011 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:34.938408 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:34.938506 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:34.940933 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:34.940937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:34.940937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:34.940950 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:34.942741 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:34.943057 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:34.943060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:34.943060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:34.943165 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:34.943382 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:34.943383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:34.943383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:34.943704 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:34.943870 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:34.944063 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 21:56:34.944197 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260925 21:56:34.944541 10566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:34.944864 10566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:34.946201 10566 catalog_manager.cc:1347] Generated new cluster ID: 18d9c73ebbb34a34a8cc9667466227ba
I20260925 21:56:34.946250 10566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:34.953821 10566 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:34.954421 10566 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:34.957243 10566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: Generated new TSK 0
I20260925 21:56:34.957401 10566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:56:34.990355 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:40965
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:35.072405 10570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:35.072626 10570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:35.072650 10570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:35.073804 10570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:35.073840 10570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:35.073865 10570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:35.075412 10570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:40965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:35.075630 10570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:35.075820 10570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:35.080677 10575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:35.080677 10578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:35.080695 10576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:35.081079 10570 server_base.cc:1034] running on GCE node
I20260925 21:56:35.081301 10570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:35.081537 10570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:35.082684 10570 hybrid_clock.cc:648] HybridClock initialized: now 1790373395082669 us; error 30 us; skew 500 ppm
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373395, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:35.089282 10570 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:56:35.089795 10570 webserver.cc:460] Webserver started at http://127.2.70.193:43101/ using document root <none> and password file <none>
I20260925 21:56:35.090034 10570 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:35.090085 10570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:35.090214 10570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:35.091295 10570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "7dc4b7c6bb1f4370883ef732b55ff883"
format_stamp: "Formatted at 2026-09-25 21:56:35 on dist-test-slave-2g0f"
server_key: "6725dca09caaa076a8c157f4a6e7248a"
server_key_iv: "4b8d052f890eaed94327ce5c3e4a4b64"
server_key_version: "encryptionkey@0"
I20260925 21:56:35.091629 10570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "7dc4b7c6bb1f4370883ef732b55ff883"
format_stamp: "Formatted at 2026-09-25 21:56:35 on dist-test-slave-2g0f"
server_key: "6725dca09caaa076a8c157f4a6e7248a"
server_key_iv: "4b8d052f890eaed94327ce5c3e4a4b64"
server_key_version: "encryptionkey@0"
I20260925 21:56:35.093107 10570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:35.093884 10585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.094086 10570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:35.094173 10570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "7dc4b7c6bb1f4370883ef732b55ff883"
format_stamp: "Formatted at 2026-09-25 21:56:35 on dist-test-slave-2g0f"
server_key: "6725dca09caaa076a8c157f4a6e7248a"
server_key_iv: "4b8d052f890eaed94327ce5c3e4a4b64"
server_key_version: "encryptionkey@0"
I20260925 21:56:35.094244 10570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:35.115605 10570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:35.117642 10570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:35.117802 10570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:35.118105 10570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:35.118413 10570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:35.118461 10570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.118490 10570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:35.118501 10570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.124972 10570 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:41085
I20260925 21:56:35.125025 10698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:41085 every 8 connection(s)
I20260925 21:56:35.125411 10570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:35.125797 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 10570
I20260925 21:56:35.125908 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:35.126101 2331 external_mini_cluster.cc:1428] Setting key 4d0ff68ab6808a5c82eb7dde8ccd0ea0
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373395, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
I20260925 21:56:35.136313 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:35.126222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:35041 (local address 127.2.70.254:40965)
0925 21:56:35.126354 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:35.126357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:35.126624 (+ 267us) server_negotiation.cc:407] Connection header received
0925 21:56:35.127276 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:35.127277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:35.127289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:35.127326 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:35.128421 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.129181 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.130212 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.130513 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.130600 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.133149 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:35.133151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:35.133151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:35.133160 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:35.134812 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:35.135270 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:35.135271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:35.135271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:35.135334 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:35.135556 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:35.135557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:35.135557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:35.135682 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:35.135739 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:35.136147 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 21:56:35.136236 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 21:56:35.136445 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:35.126231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:35041)
0925 21:56:35.126414 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:35.126499 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:35.127182 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:35.127409 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:35.127413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:35.128111 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:35.128319 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.128324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.129301 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.129302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.130130 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.130133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.130634 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.130635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.130757 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.131366 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:35.131375 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:35.133042 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:35.134926 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:35.134933 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:35.134935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:35.135180 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:35.135389 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:35.135390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:35.135390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:35.135498 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:35.135737 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:35.135743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:35.135984 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 21:56:35.136092 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 21:56:35.136249 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260925 21:56:35.136953 10699 heartbeater.cc:347] Connected to a master server at 127.2.70.254:40965
I20260925 21:56:35.137042 10699 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:35.137216 10699 heartbeater.cc:511] Master 127.2.70.254:40965 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
I20260925 21:56:35.137712 10508 ts_manager.cc:194] Registered new tserver with Master: 7dc4b7c6bb1f4370883ef732b55ff883 (127.2.70.193:41085)
I20260925 21:56:35.138542 10508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:35041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:56:35.156090 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:40965
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:35.237102 10704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:35.237324 10704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:35.237357 10704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:35.238529 10704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:35.238571 10704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:35.238597 10704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:35.240046 10704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:40965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:35.240242 10704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:35.240477 10704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:56:35.246125 10704 server_base.cc:1034] running on GCE node
W20260925 21:56:35.246021 10710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:35.246068 10709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:35.246021 10712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:35.246528 10704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:35.246786 10704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:35.247888 10704 hybrid_clock.cc:648] HybridClock initialized: now 1790373395247868 us; error 33 us; skew 500 ppm
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373395, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:35.250592 10704 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:56:35.251060 10704 webserver.cc:460] Webserver started at http://127.2.70.194:46245/ using document root <none> and password file <none>
I20260925 21:56:35.251291 10704 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:35.251341 10704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:35.251478 10704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:35.252616 10704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "08fce42ae73344ac94298d4b453f38d8"
format_stamp: "Formatted at 2026-09-25 21:56:35 on dist-test-slave-2g0f"
server_key: "d2ac06ad09927f0cce9095b735d1ad7f"
server_key_iv: "e20ffbf9253dd9cae69535363db3ac9e"
server_key_version: "encryptionkey@0"
I20260925 21:56:35.252962 10704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "08fce42ae73344ac94298d4b453f38d8"
format_stamp: "Formatted at 2026-09-25 21:56:35 on dist-test-slave-2g0f"
server_key: "d2ac06ad09927f0cce9095b735d1ad7f"
server_key_iv: "e20ffbf9253dd9cae69535363db3ac9e"
server_key_version: "encryptionkey@0"
I20260925 21:56:35.254277 10704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:35.255084 10719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.255287 10704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:35.255383 10704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "08fce42ae73344ac94298d4b453f38d8"
format_stamp: "Formatted at 2026-09-25 21:56:35 on dist-test-slave-2g0f"
server_key: "d2ac06ad09927f0cce9095b735d1ad7f"
server_key_iv: "e20ffbf9253dd9cae69535363db3ac9e"
server_key_version: "encryptionkey@0"
I20260925 21:56:35.255460 10704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:35.268636 10704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:35.270977 10704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:35.271139 10704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:35.271399 10704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:35.271727 10704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:35.271763 10704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.271793 10704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:35.271811 10704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.277364 10704 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:41857
I20260925 21:56:35.277400 10832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:41857 every 8 connection(s)
I20260925 21:56:35.277747 10704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:35.281149 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 10704
I20260925 21:56:35.281275 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:35.281461 2331 external_mini_cluster.cc:1428] Setting key f8862c8723b85526e4babf9d1ffb8755
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373395, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:35.288055 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:35.278483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:47341 (local address 127.2.70.254:40965)
0925 21:56:35.278538 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:35.278541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:35.278736 (+ 195us) server_negotiation.cc:407] Connection header received
0925 21:56:35.279248 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:35.279249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:35.279260 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:35.279303 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:35.280343 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.280843 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.282218 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.282421 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.282478 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.285096 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:35.285098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:35.285098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:35.285105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:35.286443 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:35.286923 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:35.286924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:35.286924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:35.286971 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:35.287321 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:35.287322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:35.287323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:35.287451 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:35.287517 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:35.287901 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 21:56:35.287982 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 21:56:35.288335 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:35.278451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:47341)
0925 21:56:35.278613 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:35.278656 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:35.279143 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:35.279380 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:35.279383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:35.280042 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:35.280248 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.280254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.280938 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.280940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.282134 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.282138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.282509 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.282510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.282657 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.283269 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:35.283280 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:35.284951 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:35.286533 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:35.286541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:35.286543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:35.286852 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:35.287086 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:35.287087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:35.287087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:35.287232 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:35.287520 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:35.287527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:35.287701 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 21:56:35.287822 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 21:56:35.288034 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260925 21:56:35.288905 10833 heartbeater.cc:347] Connected to a master server at 127.2.70.254:40965
I20260925 21:56:35.289011 10833 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:35.289206 10833 heartbeater.cc:511] Master 127.2.70.254:40965 requested a full tablet report, sending...
I20260925 21:56:35.289669 10508 ts_manager.cc:194] Registered new tserver with Master: 08fce42ae73344ac94298d4b453f38d8 (127.2.70.194:41857)
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
I20260925 21:56:35.290174 10508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:47341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:56:35.307672 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:40965
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:35.388797 10838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:35.389024 10838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:35.389056 10838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:35.390237 10838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:35.390280 10838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:35.390314 10838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:35.391759 10838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:40965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:35.392026 10838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:35.392236 10838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:35.397151 10843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:35.397213 10846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:35.397248 10838 server_base.cc:1034] running on GCE node
W20260925 21:56:35.397145 10844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:35.397672 10838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:35.397910 10838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:35.399072 10838 hybrid_clock.cc:648] HybridClock initialized: now 1790373395399056 us; error 30 us; skew 500 ppm
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373395, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:35.401566 10838 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:56:35.402064 10838 webserver.cc:460] Webserver started at http://127.2.70.195:43267/ using document root <none> and password file <none>
I20260925 21:56:35.402292 10838 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:35.402335 10838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:35.402450 10838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:35.403451 10838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "bed0e04fd7cc4fdcadca06a60b55e524"
format_stamp: "Formatted at 2026-09-25 21:56:35 on dist-test-slave-2g0f"
server_key: "12095d835df78ecc83265361ccff7924"
server_key_iv: "6e909593dc7f7f923b1bc73f1758a1fa"
server_key_version: "encryptionkey@0"
I20260925 21:56:35.403730 10838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "bed0e04fd7cc4fdcadca06a60b55e524"
format_stamp: "Formatted at 2026-09-25 21:56:35 on dist-test-slave-2g0f"
server_key: "12095d835df78ecc83265361ccff7924"
server_key_iv: "6e909593dc7f7f923b1bc73f1758a1fa"
server_key_version: "encryptionkey@0"
I20260925 21:56:35.405010 10838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:35.405649 10853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.405802 10838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 21:56:35.405947 10838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "bed0e04fd7cc4fdcadca06a60b55e524"
format_stamp: "Formatted at 2026-09-25 21:56:35 on dist-test-slave-2g0f"
server_key: "12095d835df78ecc83265361ccff7924"
server_key_iv: "6e909593dc7f7f923b1bc73f1758a1fa"
server_key_version: "encryptionkey@0"
I20260925 21:56:35.406013 10838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:35.428663 10838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:35.430546 10838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:35.430713 10838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:35.430944 10838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:35.431236 10838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:35.431267 10838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.431296 10838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:35.431320 10838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.436764 10838 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:44591
I20260925 21:56:35.436815 10966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:44591 every 8 connection(s)
I20260925 21:56:35.437119 10838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:35.442803 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 10838
I20260925 21:56:35.442912 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:35.443089 2331 external_mini_cluster.cc:1428] Setting key 382377a977dda4e6a90c794be6d5530e
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373395, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:35.446398 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:35.437783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:35905 (local address 127.2.70.254:40965)
0925 21:56:35.437837 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:35.437839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:35.438040 (+ 201us) server_negotiation.cc:407] Connection header received
0925 21:56:35.438464 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:35.438465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:35.438473 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:35.438539 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:35.439437 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.439959 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.440954 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.441207 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.441255 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.443819 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:35.443820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:35.443821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:35.443847 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:35.445205 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:35.445594 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:35.445596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:35.445596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:35.445655 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:35.445851 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:35.445851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:35.445851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:35.445954 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:35.446003 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:35.446256 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 21:56:35.446330 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 21:56:35.446530 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:35.437737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:35905)
0925 21:56:35.437892 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:35.437935 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:35.438392 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:35.438596 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:35.438600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:35.439161 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:35.439361 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.439366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.440036 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.440037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.440886 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.440888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.441278 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.441279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.441388 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.442016 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:35.442025 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:35.443724 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:35.445279 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:35.445285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:35.445287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:35.445514 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:35.445710 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:35.445711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:35.445711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:35.445803 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:35.445998 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:35.446000 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:35.446156 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 21:56:35.446238 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 21:56:35.446362 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260925 21:56:35.446981 10967 heartbeater.cc:347] Connected to a master server at 127.2.70.254:40965
I20260925 21:56:35.447083 10967 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:35.447228 10967 heartbeater.cc:511] Master 127.2.70.254:40965 requested a full tablet report, sending...
I20260925 21:56:35.447583 10508 ts_manager.cc:194] Registered new tserver with Master: bed0e04fd7cc4fdcadca06a60b55e524 (127.2.70.195:44591)
I20260925 21:56:35.448063 10508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:35905
I20260925 21:56:35.455014 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab.
Entry for principal oryx/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab.
WARNING: no policy specified for oryx/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab.
Entry for principal oryx/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab.
WARNING: no policy specified for oryx/127.2.70.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab.
Entry for principal oryx/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab.
I20260925 21:56:35.526743 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 10570
I20260925 21:56:35.531175 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 10704
I20260925 21:56:35.535303 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 10838
I20260925 21:56:35.539187 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 10476
I20260925 21:56:35.543708 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:40965
--webserver_interface=127.2.70.254
--webserver_port=38615
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:40965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:35.625207 10976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:35.625439 10976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:35.625463 10976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:35.626653 10976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:35.626700 10976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:35.626713 10976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:35.626725 10976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:35.626735 10976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:35.628279 10976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:40965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:40965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:35.628531 10976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:35.628749 10976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:35.633520 10981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:35.633515 10982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:35.633554 10976 server_base.cc:1034] running on GCE node
W20260925 21:56:35.633556 10984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:35.633986 10976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:35.634258 10976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:35.635432 10976 hybrid_clock.cc:648] HybridClock initialized: now 1790373395635414 us; error 32 us; skew 500 ppm
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373395, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:35.637908 10976 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:56:35.638387 10976 webserver.cc:460] Webserver started at http://127.2.70.254:38615/ using document root <none> and password file <none>
I20260925 21:56:35.638604 10976 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:35.638664 10976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:35.640004 10976 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:35.640681 10991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.640882 10976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:35.640970 10976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "dca47c537e884fec9e18a8e1a212d4b2"
format_stamp: "Formatted at 2026-09-25 21:56:34 on dist-test-slave-2g0f"
server_key: "fc2738273692a89a2e9fc8928b26af77"
server_key_iv: "88254206e9404a0392527e0d5d266f39"
server_key_version: "encryptionkey@0"
I20260925 21:56:35.641336 10976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:35.649420 10976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:35.651620 10976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:35.651760 10976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:35.655542 10976 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:40965
I20260925 21:56:35.655575 11043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:40965 every 8 connection(s)
I20260925 21:56:35.656009 10976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:35.657464 11044 sys_catalog.cc:263] Verifying existing consensus state
I20260925 21:56:35.658178 11044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: Bootstrap starting.
I20260925 21:56:35.658447 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 10976
I20260925 21:56:35.658771 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:41085
--local_ip_for_outbound_sockets=127.2.70.193
--tserver_master_addrs=127.2.70.254:40965
--webserver_port=43101
--webserver_interface=127.2.70.193
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.70.193
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:56:35.666590 11044 log.cc:824] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:35.668524 11044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 21:56:35.668798 11044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: Bootstrap complete.
I20260925 21:56:35.670948 11044 raft_consensus.cc:348] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca47c537e884fec9e18a8e1a212d4b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40965 } }
I20260925 21:56:35.671183 11044 raft_consensus.cc:729] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca47c537e884fec9e18a8e1a212d4b2, State: Initialized, Role: FOLLOWER
I20260925 21:56:35.671346 11044 consensus_queue.cc:260] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca47c537e884fec9e18a8e1a212d4b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40965 } }
I20260925 21:56:35.671415 11044 raft_consensus.cc:388] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:35.671464 11044 raft_consensus.cc:482] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:35.671512 11044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 1 FOLLOWER]: Advancing to term 2
I20260925 21:56:35.672641 11044 raft_consensus.cc:504] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca47c537e884fec9e18a8e1a212d4b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40965 } }
I20260925 21:56:35.672735 11044 leader_election.cc:302] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dca47c537e884fec9e18a8e1a212d4b2; no voters:
I20260925 21:56:35.672902 11044 leader_election.cc:288] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 21:56:35.672960 11048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 21:56:35.673123 11048 raft_consensus.cc:686] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [term 2 LEADER]: Becoming Leader. State: Replica: dca47c537e884fec9e18a8e1a212d4b2, State: Running, Role: LEADER
I20260925 21:56:35.673185 11044 sys_catalog.cc:564] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:35.673635 11048 consensus_queue.cc:237] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca47c537e884fec9e18a8e1a212d4b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40965 } }
I20260925 21:56:35.674036 11050 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dca47c537e884fec9e18a8e1a212d4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca47c537e884fec9e18a8e1a212d4b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40965 } } }
I20260925 21:56:35.674041 11049 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dca47c537e884fec9e18a8e1a212d4b2. Latest consensus state: current_term: 2 leader_uuid: "dca47c537e884fec9e18a8e1a212d4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca47c537e884fec9e18a8e1a212d4b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40965 } } }
I20260925 21:56:35.674293 11049 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:35.674502 11050 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:35.674924 11064 catalog_manager.cc:1259] Loaded cluster ID: 18d9c73ebbb34a34a8cc9667466227ba
I20260925 21:56:35.674961 11064 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: loading cluster ID for follower catalog manager: success
I20260925 21:56:35.675733 11064 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: acquiring CA information for follower catalog manager: success
I20260925 21:56:35.676080 11064 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 21:56:35.676184 11063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:35.676326 11063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:35.676371 11063 catalog_manager.cc:1259] Loaded cluster ID: 18d9c73ebbb34a34a8cc9667466227ba
I20260925 21:56:35.676384 11063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:35.676493 11063 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:35.676542 11063 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dca47c537e884fec9e18a8e1a212d4b2: Loaded TSK: 0
I20260925 21:56:35.676615 11063 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 21:56:35.747292 11046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:35.747520 11046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:35.747556 11046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:35.748867 11046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:35.748920 11046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:35.748960 11046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:35.750520 11046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.193.keytab
--principal=oryx/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:41085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=43101
--tserver_master_addrs=127.2.70.254:40965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:35.750794 11046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:35.751024 11046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:35.756135 11070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:35.756196 11073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:35.756173 11071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:35.756567 11046 server_base.cc:1034] running on GCE node
I20260925 21:56:35.756774 11046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:35.757027 11046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:35.758221 11046 hybrid_clock.cc:648] HybridClock initialized: now 1790373395758193 us; error 32 us; skew 500 ppm
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373395, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:35.761173 11046 init.cc:377] Logged in from keytab as oryx/127.2.70.193@KRBTEST.COM (short username oryx)
I20260925 21:56:35.761814 11046 webserver.cc:460] Webserver started at http://127.2.70.193:43101/ using document root <none> and password file <none>
I20260925 21:56:35.762120 11046 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:35.762216 11046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:35.764213 11046 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:35.764971 11080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.765144 11046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:56:35.765244 11046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "7dc4b7c6bb1f4370883ef732b55ff883"
format_stamp: "Formatted at 2026-09-25 21:56:35 on dist-test-slave-2g0f"
server_key: "6725dca09caaa076a8c157f4a6e7248a"
server_key_iv: "4b8d052f890eaed94327ce5c3e4a4b64"
server_key_version: "encryptionkey@0"
I20260925 21:56:35.765704 11046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:35.791401 11046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:35.794334 11046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:35.794498 11046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:35.794778 11046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:35.795152 11046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:35.795202 11046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.795234 11046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:35.795252 11046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.802183 11046 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:41085
I20260925 21:56:35.802268 11193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:41085 every 8 connection(s)
I20260925 21:56:35.802637 11046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:35.804396 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 11046
I20260925 21:56:35.804682 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:41857
--local_ip_for_outbound_sockets=127.2.70.194
--tserver_master_addrs=127.2.70.254:40965
--webserver_port=46245
--webserver_interface=127.2.70.194
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.70.194
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:35.813357 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:35.803439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:53591)
0925 21:56:35.803612 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:35.803656 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:35.804297 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:35.804762 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:35.804766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:35.805355 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:35.805566 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.805577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.807239 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.807240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.808547 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.808551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.809417 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.809419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.809580 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.810179 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:35.810190 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:35.813119 (+ 2929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
W20260925 21:56:35.813493 11197 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:35.803564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:53591 (local address 127.2.70.254:40965)
0925 21:56:35.803755 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:35.803763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:35.803804 (+ 41us) server_negotiation.cc:407] Connection header received
0925 21:56:35.804539 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:35.804545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:35.804593 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:35.804715 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:35.805721 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.807143 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.808656 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.809288 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.809413 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.813287 (+ 3874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:53591: BlockingRecv error: recv got EOF from 127.2.70.193:53591 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
W20260925 21:56:35.813927 11194 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:40965 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.70.254:40965: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:35.819106 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:35.814171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:59359)
0925 21:56:35.814219 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:35.814230 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:35.814269 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:35.814406 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:35.814407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:35.814551 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:35.814614 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.814616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.815440 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.815441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.816334 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.816336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.816819 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.816820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.816892 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.817284 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:35.817289 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:35.819052 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 21:56:35.819446 11197 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:35.814217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:59359 (local address 127.2.70.254:40965)
0925 21:56:35.814250 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:35.814252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:35.814260 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:35.814325 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:35.814325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:35.814337 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:35.814389 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:35.814684 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.815356 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.816427 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.816746 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.816818 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.819230 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:59359: BlockingRecv error: recv got EOF from 127.2.70.193:59359 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:35.823398 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:35.819361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:40029)
0925 21:56:35.819410 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:35.819421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:35.819465 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:35.819706 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:35.819707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:35.819823 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:35.819892 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.819894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.820552 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.820553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.821046 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.821047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.821364 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.821365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.821421 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.821799 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:35.821804 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:35.823327 (+ 1523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 21:56:35.823482 11199 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:35.819417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:40029 (local address 127.2.70.254:40965)
0925 21:56:35.819580 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:35.819582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:35.819588 (+ 6us) server_negotiation.cc:407] Connection header received
0925 21:56:35.819608 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:35.819608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:35.819617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:35.819698 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:35.819957 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.820475 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.821113 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.821309 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.821362 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.823417 (+ 2055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:40029: BlockingRecv error: recv got EOF from 127.2.70.193:40029 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":112,"threads_started":1}
W20260925 21:56:35.823621 11194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 21:56:35.889009 11198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:35.889228 11198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:35.889259 11198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:35.890496 11198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:35.890537 11198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:35.890586 11198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:35.892086 11198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.194.keytab
--principal=oryx/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:41857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=46245
--tserver_master_addrs=127.2.70.254:40965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:35.892350 11198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:35.892572 11198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:35.897485 11205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:35.897526 11204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:35.897501 11207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:35.897876 11198 server_base.cc:1034] running on GCE node
I20260925 21:56:35.898096 11198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:35.898304 11198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:35.899480 11198 hybrid_clock.cc:648] HybridClock initialized: now 1790373395899468 us; error 31 us; skew 500 ppm
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373395, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:35.901966 11198 init.cc:377] Logged in from keytab as oryx/127.2.70.194@KRBTEST.COM (short username oryx)
I20260925 21:56:35.902441 11198 webserver.cc:460] Webserver started at http://127.2.70.194:46245/ using document root <none> and password file <none>
I20260925 21:56:35.902665 11198 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:35.902729 11198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:35.904100 11198 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:35.904798 11214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.904999 11198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:35.905099 11198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "08fce42ae73344ac94298d4b453f38d8"
format_stamp: "Formatted at 2026-09-25 21:56:35 on dist-test-slave-2g0f"
server_key: "d2ac06ad09927f0cce9095b735d1ad7f"
server_key_iv: "e20ffbf9253dd9cae69535363db3ac9e"
server_key_version: "encryptionkey@0"
I20260925 21:56:35.905460 11198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:35.915434 11198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:35.917377 11198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:35.917520 11198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:35.917743 11198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:35.918054 11198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:35.918088 11198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.918118 11198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:35.918141 11198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:35.923609 11198 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:41857
I20260925 21:56:35.923648 11327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:41857 every 8 connection(s)
I20260925 21:56:35.924041 11198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:35.930650 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 11198
I20260925 21:56:35.930897 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:44591
--local_ip_for_outbound_sockets=127.2.70.195
--tserver_master_addrs=127.2.70.254:40965
--webserver_port=43267
--webserver_interface=127.2.70.195
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.70.195
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:56:35.931216 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:35.924730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:34049)
0925 21:56:35.924882 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:35.924928 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:35.925424 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:35.925656 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:35.925660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:35.926245 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:35.926447 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.926452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.927124 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.927125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.927984 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.927987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.928325 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.928326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.928469 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.928931 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:35.928940 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:35.931029 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
W20260925 21:56:35.931248 11199 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:35.924737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:34049 (local address 127.2.70.254:40965)
0925 21:56:35.924797 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:35.924798 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:35.925019 (+ 221us) server_negotiation.cc:407] Connection header received
0925 21:56:35.925517 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:35.925519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:35.925533 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:35.925584 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:35.926541 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.927043 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.928060 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.928258 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.928315 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.931187 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:34049: BlockingRecv error: recv got EOF from 127.2.70.194:34049 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 21:56:35.931514 11328 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:40965 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.70.254:40965: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:35.936125 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:35.931723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:43023)
0925 21:56:35.931782 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:35.931792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:35.931851 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:35.931969 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:35.931970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:35.932095 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:35.932144 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.932146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.932756 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.932757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.933403 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.933406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.933803 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.933804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.933865 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.934286 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:35.934291 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:35.936054 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 21:56:35.936195 11199 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:35.931778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:43023 (local address 127.2.70.254:40965)
0925 21:56:35.931859 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:35.931861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:35.931872 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:56:35.931902 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:35.931903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:35.931915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:35.931960 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:35.932210 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.932688 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.933511 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.933724 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.933808 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.936141 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:43023: BlockingRecv error: recv got EOF from 127.2.70.194:43023 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:35 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:35.940668 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:35.936632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:36203)
0925 21:56:35.936693 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:35.936703 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:35.936737 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:35.936860 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:35.936861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:35.936978 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:35.937007 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.937009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.937610 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.937611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.938168 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:35.938170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.938506 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.938507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:35.938556 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.938992 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:35.938996 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:35.940599 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 21:56:35.940796 11199 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:35.936706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:36203 (local address 127.2.70.254:40965)
0925 21:56:35.936762 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:35.936764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:35.936772 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:35.936794 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:35.936795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:35.936807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:35.936853 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:35.937067 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.937547 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.938245 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:35.938449 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:35.938502 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:35.940686 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:36203: BlockingRecv error: recv got EOF from 127.2.70.194:36203 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 21:56:35.940799 11328 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 21:56:36.012889 11331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:36.013128 11331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:36.013154 11331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:36.014393 11331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:36.014438 11331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:36.014464 11331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:36.015986 11331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/krb5kdc/oryx_127.2.70.195.keytab
--principal=oryx/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:44591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=43267
--tserver_master_addrs=127.2.70.254:40965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:36.016245 11331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:36.016474 11331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:36.021497 11336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:36.021479 11339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:36.021661 11331 server_base.cc:1034] running on GCE node
W20260925 21:56:36.021481 11337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:36.021939 11331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:36.022195 11331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:36.023335 11331 hybrid_clock.cc:648] HybridClock initialized: now 1790373396023317 us; error 36 us; skew 500 ppm
Sep 25 21:56:36 dist-test-slave-2g0f krb5kdc[10463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373396, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:36.025811 11331 init.cc:377] Logged in from keytab as oryx/127.2.70.195@KRBTEST.COM (short username oryx)
I20260925 21:56:36.026268 11331 webserver.cc:460] Webserver started at http://127.2.70.195:43267/ using document root <none> and password file <none>
I20260925 21:56:36.026496 11331 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:36.026568 11331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:36.028021 11331 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:36.028565 11346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:36.028709 11331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:36.028791 11331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "bed0e04fd7cc4fdcadca06a60b55e524"
format_stamp: "Formatted at 2026-09-25 21:56:35 on dist-test-slave-2g0f"
server_key: "12095d835df78ecc83265361ccff7924"
server_key_iv: "6e909593dc7f7f923b1bc73f1758a1fa"
server_key_version: "encryptionkey@0"
I20260925 21:56:36.029120 11331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:36.044346 11331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:36.046643 11331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:36.046789 11331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:36.047010 11331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:36.047324 11331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:36.047358 11331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:36.047387 11331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:36.047410 11331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:36.054287 11331 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:44591
I20260925 21:56:36.054356 11459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:44591 every 8 connection(s)
I20260925 21:56:36.054708 11331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:36.056255 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 11331
Sep 25 21:56:36 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:36 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 21:56:36.063371 11199 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:36.055491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:56375 (local address 127.2.70.254:40965)
0925 21:56:36.055591 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:36.055593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:36.055760 (+ 167us) server_negotiation.cc:407] Connection header received
0925 21:56:36.056771 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:36.056772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:36.056786 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:36.056849 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:36.057993 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.058611 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.059923 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.060107 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.060159 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:36.063281 (+ 3122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:56375: BlockingRecv error: recv got EOF from 127.2.70.195:56375 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 21:56:36.063411 11462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:36.055420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:56375)
0925 21:56:36.055627 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:36.055673 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:36.056389 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:36.056955 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:36.057025 (+ 70us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:36.057622 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:36.057887 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:36.057893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.058726 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.058727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:36.059809 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:36.059812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.060175 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.060176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:36.060296 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:36.060856 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:36.060864 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:36.063181 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
W20260925 21:56:36.063627 11460 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:40965 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.70.254:40965: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
I20260925 21:56:36.064220 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:36.056821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.2.70.254:40965)
0925 21:56:36.056869 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:36.056871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:36.056900 (+ 29us) server_negotiation.cc:407] Connection header received
0925 21:56:36.056920 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:36.056921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:36.056933 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:36.056987 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:36.057580 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.058218 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.059114 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.059416 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.059496 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:36.060828 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:36.060833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:36.060834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:36.060848 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:36.062661 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:36.063006 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:36.063009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:36.063010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:36.063117 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:36.063287 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:36.063288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:36.063288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:36.063624 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:36.063755 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:36.063993 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 21:56:36.064131 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 21:56:36 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:36 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:36.068173 11462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:36.063876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:37097)
0925 21:56:36.063922 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:36.063933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:36.063964 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:36.064114 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:36.064115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:36.064233 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:36.064288 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:36.064291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.065075 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.065076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:36.065624 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:36.065627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.065929 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.065929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:36.065975 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:36.066357 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:36.066362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:36.068075 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 21:56:36.068228 11199 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:36.063912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:37097 (local address 127.2.70.254:40965)
0925 21:56:36.063953 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:36.063954 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:36.063962 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:36.064021 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:36.064021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:36.064031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:36.064079 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:36.064357 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.064951 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.065687 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.065874 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.065926 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:36.068175 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:37097: BlockingRecv error: recv got EOF from 127.2.70.195:37097 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 21:56:36 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:36 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:36.072638 11462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:36.068469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:59245)
0925 21:56:36.068516 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:36.068528 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:36.068576 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:36.068692 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:36.068692 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:36.068801 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:36.068833 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:36.068834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.069442 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.069443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:36.069965 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:36.069967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.070355 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.070356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:36.070396 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:36.070830 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:36.070834 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:36.072545 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 21:56:36.072710 11199 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:36.068542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:59245 (local address 127.2.70.254:40965)
0925 21:56:36.068596 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:36.068598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:36.068618 (+ 20us) server_negotiation.cc:407] Connection header received
0925 21:56:36.068637 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:36.068638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:36.068646 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:36.068681 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:36.068880 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.069385 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.070107 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.070300 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.070354 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:36.072649 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:59245: BlockingRecv error: recv got EOF from 127.2.70.195:59245 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260925 21:56:36.072784 11460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 21:56:36 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:36 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:36.828588 11464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:36.824032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:35265)
0925 21:56:36.824160 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:36.824171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:36.824219 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:36.824421 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:36.824422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:36.824576 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:36.824624 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:36.824626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.825277 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.825277 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:36.825824 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:36.825825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.826222 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.826224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:36.826271 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:36.826717 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:36.826722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:36.828515 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
W20260925 21:56:36.828675 11465 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:36.824195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:35265 (local address 127.2.70.254:40965)
0925 21:56:36.824310 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:36.824312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:36.824319 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:56:36.824340 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:36.824340 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:36.824351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:36.824412 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:36.824677 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.825215 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.825905 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.826146 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.826208 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:36.828610 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:35265: BlockingRecv error: recv got EOF from 127.2.70.193:35265 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
Sep 25 21:56:36 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:36 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:36.945797 11466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:36.941339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:41731)
0925 21:56:36.941509 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:36.941521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:36.941588 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:36.941709 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:36.941710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:36.941866 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:36.941914 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:36.941915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.942547 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.942548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:36.943111 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:36.943113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.943521 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.943523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:36.943586 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:36.944083 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:36.944088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:36.945729 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
W20260925 21:56:36.945883 11465 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:36.941435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:41731 (local address 127.2.70.254:40965)
0925 21:56:36.941502 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:36.941504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:36.941580 (+ 76us) server_negotiation.cc:407] Connection header received
0925 21:56:36.941634 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:36.941635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:36.941646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:36.941695 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:36.941975 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.942462 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.943205 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:36.943422 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:36.943509 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:36.945823 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:41731: BlockingRecv error: recv got EOF from 127.2.70.194:41731 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 21:56:37 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:37 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:37.077741 11467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:37.073241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:48275)
0925 21:56:37.073391 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:37.073403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:37.073476 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:37.073671 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:37.073672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:37.073871 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:37.073920 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:37.073922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:37.074689 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:37.074690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:37.075249 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:37.075251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:37.075574 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:37.075575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:37.075610 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:37.076021 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:37.076025 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:37.077618 (+ 1593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
W20260925 21:56:37.077747 11465 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:37.073302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:48275 (local address 127.2.70.254:40965)
0925 21:56:37.073358 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:37.073361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:37.073448 (+ 87us) server_negotiation.cc:407] Connection header received
0925 21:56:37.073582 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:37.073584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:37.073598 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:37.073656 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:37.074065 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:37.074604 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:37.075329 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:37.075519 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:37.075573 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:37.077699 (+ 2126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:48275: BlockingRecv error: recv got EOF from 127.2.70.195:48275 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 21:56:37 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:37 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:37.833529 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:37.829027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:46031)
0925 21:56:37.829168 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:37.829177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:37.829231 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:37.829357 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:37.829358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:37.829524 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:37.829572 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:37.829573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:37.830307 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:37.830308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:37.830853 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:37.830855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:37.831197 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:37.831198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:37.831241 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:37.831713 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:37.831718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:37.833435 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
W20260925 21:56:37.833599 11469 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:37.829120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:46031 (local address 127.2.70.254:40965)
0925 21:56:37.829225 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:37.829227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:37.829236 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:37.829277 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:37.829278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:37.829288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:37.829347 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:37.829658 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:37.830235 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:37.830936 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:37.831130 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:37.831183 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:37.833529 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:46031: BlockingRecv error: recv got EOF from 127.2.70.193:46031 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
Sep 25 21:56:37 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:37 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:37.950717 11470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:37.946288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:58381)
0925 21:56:37.946426 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:37.946439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:37.946521 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:37.946661 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:37.946662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:37.946862 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:37.946946 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:37.946949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:37.947607 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:37.947608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:37.948258 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:37.948260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:37.948590 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:37.948591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:37.948644 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:37.949070 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:37.949075 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:37.950651 (+ 1576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
W20260925 21:56:37.950807 11469 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:37.946372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:58381 (local address 127.2.70.254:40965)
0925 21:56:37.946464 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:37.946466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:37.946501 (+ 35us) server_negotiation.cc:407] Connection header received
0925 21:56:37.946587 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:37.946588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:37.946600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:37.946652 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:37.947010 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:37.947532 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:37.948337 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:37.948533 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:37.948587 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:37.950749 (+ 2162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:58381: BlockingRecv error: recv got EOF from 127.2.70.194:58381 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 21:56:38 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:38 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:38.082394 11471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:38.078195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:34745)
0925 21:56:38.078361 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:38.078375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:38.078439 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:38.078559 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:38.078560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:38.078727 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:38.078771 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:38.078773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:38.079390 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:38.079390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:38.079981 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:38.079984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:38.080293 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:38.080294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:38.080330 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:38.080735 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:38.080739 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:38.082325 (+ 1586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
W20260925 21:56:38.082460 11469 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:38.078341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:34745 (local address 127.2.70.254:40965)
0925 21:56:38.078387 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:38.078389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:38.078422 (+ 33us) server_negotiation.cc:407] Connection header received
0925 21:56:38.078490 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:38.078490 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:38.078501 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:38.078538 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:38.078829 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:38.079313 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:38.080051 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:38.080238 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:38.080289 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:38.082407 (+ 2118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:34745: BlockingRecv error: recv got EOF from 127.2.70.195:34745 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 21:56:38 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:38 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:38.838706 11472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:38.834002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:56341)
0925 21:56:38.834177 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:38.834189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:38.834259 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:38.834390 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:38.834391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:38.834551 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:38.834598 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:38.834600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:38.835234 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:38.835235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:38.836055 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:38.836058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:38.836435 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:38.836436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:38.836478 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:38.836883 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:38.836932 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:38.838635 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
W20260925 21:56:38.838822 11473 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:38.834082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:56341 (local address 127.2.70.254:40965)
0925 21:56:38.834203 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:38.834205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:38.834253 (+ 48us) server_negotiation.cc:407] Connection header received
0925 21:56:38.834305 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:38.834306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:38.834317 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:38.834375 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:38.834650 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:38.835158 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:38.836145 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:38.836362 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:38.836422 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:38.838752 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:56341: BlockingRecv error: recv got EOF from 127.2.70.193:56341 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
Sep 25 21:56:38 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:38 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:38.955585 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:38.951184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:46807)
0925 21:56:38.951312 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:38.951325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:38.951393 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:38.951532 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:38.951533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:38.951705 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:38.951750 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:38.951752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:38.952431 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:38.952432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:38.952980 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:38.952982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:38.953317 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:38.953318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:38.953373 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:38.953799 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:38.953804 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:38.955516 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
W20260925 21:56:38.955679 11473 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:38.951295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:46807 (local address 127.2.70.254:40965)
0925 21:56:38.951343 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:38.951346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:38.951375 (+ 29us) server_negotiation.cc:407] Connection header received
0925 21:56:38.951448 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:38.951449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:38.951462 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:38.951506 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:38.951852 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:38.952357 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:38.953066 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:38.953258 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:38.953313 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:38.955616 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:46807: BlockingRecv error: recv got EOF from 127.2.70.194:46807 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 21:56:39 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:39 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:39.087414 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:39.082888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:49015)
0925 21:56:39.083043 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:39.083055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:39.083117 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:39.083241 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:39.083242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:39.083414 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:39.083466 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:39.083468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:39.084141 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:39.084142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:39.084722 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:39.084724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:39.085106 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:39.085107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:39.085160 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:39.085644 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:39.085656 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:39.087341 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
W20260925 21:56:39.087544 11473 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:39.082978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:49015 (local address 127.2.70.254:40965)
0925 21:56:39.083063 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:39.083064 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:39.083111 (+ 47us) server_negotiation.cc:407] Connection header received
0925 21:56:39.083170 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:39.083171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:39.083183 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:39.083233 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:39.083520 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:39.084067 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:39.084803 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:39.084990 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:39.085069 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:39.087470 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:49015: BlockingRecv error: recv got EOF from 127.2.70.195:49015 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 21:56:39 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:39 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:39.843631 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:39.839221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:36383)
0925 21:56:39.839340 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:39.839351 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:39.839410 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:39.839559 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:39.839560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:39.839711 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:39.839760 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:39.839761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:39.840457 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:39.840458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:39.841030 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:39.841032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:39.841410 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:39.841410 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:39.841452 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:39.841938 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:39.841943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:39.843569 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
W20260925 21:56:39.843746 11477 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:39.839308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:36383 (local address 127.2.70.254:40965)
0925 21:56:39.839422 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:39.839424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:39.839447 (+ 23us) server_negotiation.cc:407] Connection header received
0925 21:56:39.839471 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:39.839472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:39.839482 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:39.839530 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:39.839814 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:39.840326 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:39.841142 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:39.841345 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:39.841402 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:39.843666 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:36383: BlockingRecv error: recv got EOF from 127.2.70.193:36383 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":48,"threads_started":1}
Sep 25 21:56:39 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:39 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:39.960876 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:39.956095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:38701)
0925 21:56:39.956238 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:39.956249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:39.956307 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:39.956428 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:39.956429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:39.956590 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:39.956637 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:39.956639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:39.957365 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:39.957366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:39.957976 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:39.957979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:39.958373 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:39.958374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:39.958422 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:39.958948 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:39.958953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:39.960797 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
W20260925 21:56:39.960966 11477 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:39.956192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:38701 (local address 127.2.70.254:40965)
0925 21:56:39.956263 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:39.956264 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:39.956304 (+ 40us) server_negotiation.cc:407] Connection header received
0925 21:56:39.956351 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:39.956352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:39.956364 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:39.956419 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:39.956698 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:39.957274 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:39.958094 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:39.958299 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:39.958353 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:39.960912 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:38701: BlockingRecv error: recv got EOF from 127.2.70.194:38701 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 21:56:40 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:40 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:40.096030 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:40.087958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:54721)
0925 21:56:40.088101 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:40.088113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:40.088191 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:40.088320 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:40.088321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:40.088482 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:40.088527 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:40.088529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:40.089155 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:40.089156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:40.089689 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:40.089691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:40.090042 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:40.090043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:40.090098 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:40.094068 (+ 3970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:40.094074 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:40.095943 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
W20260925 21:56:40.096088 11477 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:40.088018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:54721 (local address 127.2.70.254:40965)
0925 21:56:40.088105 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:40.088107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:40.088182 (+ 75us) server_negotiation.cc:407] Connection header received
0925 21:56:40.088245 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:40.088246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:40.088259 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:40.088308 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:40.088601 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:40.089086 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:40.089763 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:40.089975 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:40.090032 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:40.096033 (+ 6001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:54721: BlockingRecv error: recv got EOF from 127.2.70.195:54721 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 21:56:40 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:40 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:40.848696 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:40.844140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:43643)
0925 21:56:40.844265 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:40.844274 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:40.844325 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:40.844485 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:40.844486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:40.844640 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:40.844688 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:40.844689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:40.845457 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:40.845458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:40.845989 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:40.845991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:40.846337 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:40.846337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:40.846374 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:40.846866 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:40.846870 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:40.848631 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
W20260925 21:56:40.848866 11481 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:40.844207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:43643 (local address 127.2.70.254:40965)
0925 21:56:40.844325 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:40.844327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:40.844348 (+ 21us) server_negotiation.cc:407] Connection header received
0925 21:56:40.844375 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:40.844376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:40.844387 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:40.844458 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:40.844772 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:40.845386 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:40.846078 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:40.846278 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:40.846334 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:40.848787 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:43643: BlockingRecv error: recv got EOF from 127.2.70.193:43643 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
Sep 25 21:56:40 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:40 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:40.966125 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:40.961426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:50457)
0925 21:56:40.961625 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:40.961638 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:40.961716 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:40.961838 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:40.961838 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:40.961996 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:40.962044 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:40.962046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:40.962715 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:40.962717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:40.963343 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:40.963345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:40.963735 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:40.963737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:40.963789 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:40.964279 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:40.964284 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:40.966047 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":105,"threads_started":1}
W20260925 21:56:40.966203 11481 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:40.961591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:50457 (local address 127.2.70.254:40965)
0925 21:56:40.961644 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:40.961646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:40.961692 (+ 46us) server_negotiation.cc:407] Connection header received
0925 21:56:40.961760 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:40.961761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:40.961771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:40.961817 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:40.962112 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:40.962631 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:40.963436 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:40.963633 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:40.963696 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:40.966151 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:50457: BlockingRecv error: recv got EOF from 127.2.70.194:50457 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 21:56:41 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:41 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:41.100965 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:41.096532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:58039)
0925 21:56:41.096706 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:41.096719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:41.096798 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:41.096973 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:41.096974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:41.097135 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:41.097181 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:41.097182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:41.097840 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:41.097842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:41.098368 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:41.098370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:41.098692 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:41.098693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:41.098746 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:41.099227 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:41.099231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:41.100902 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
W20260925 21:56:41.101055 11481 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:41.096624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:58039 (local address 127.2.70.254:40965)
0925 21:56:41.096710 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:41.096712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:41.096777 (+ 65us) server_negotiation.cc:407] Connection header received
0925 21:56:41.096852 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:41.096853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:41.096866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:41.096917 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:41.097256 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:41.097767 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:41.098449 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:41.098635 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:41.098689 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:41.100993 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:58039: BlockingRecv error: recv got EOF from 127.2.70.195:58039 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 21:56:41 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:41 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:41.853494 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:41.849173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:39827)
0925 21:56:41.849291 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:41.849302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:41.849363 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:41.849500 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:41.849501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:41.849664 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:41.849710 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:41.849712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:41.850379 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:41.850380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:41.850948 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:41.850950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:41.851275 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:41.851276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:41.851324 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:41.851759 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:41.851763 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:41.853434 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
W20260925 21:56:41.853624 11485 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:41.849255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:39827 (local address 127.2.70.254:40965)
0925 21:56:41.849360 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:41.849362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:41.849371 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:41.849412 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:41.849413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:41.849431 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:41.849487 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:41.849764 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:41.850278 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:41.851014 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:41.851205 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:41.851253 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:41.853544 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:39827: BlockingRecv error: recv got EOF from 127.2.70.193:39827 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":49,"threads_started":1}
Sep 25 21:56:41 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:41 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:41.971186 11486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:41.966609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:50077)
0925 21:56:41.966721 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:41.966733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:41.966781 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:41.966961 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:41.966962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:41.967123 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:41.967170 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:41.967171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:41.967872 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:41.967873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:41.968425 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:41.968427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:41.968736 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:41.968737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:41.968773 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:41.969250 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:41.969254 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:41.971110 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
W20260925 21:56:41.971257 11485 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:41.966694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:50077 (local address 127.2.70.254:40965)
0925 21:56:41.966763 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:41.966765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:41.966774 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:41.966827 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:41.966828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:41.966837 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:41.966945 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:41.967251 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:41.967753 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:41.968496 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:41.968682 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:41.968735 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:41.971199 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:50077: BlockingRecv error: recv got EOF from 127.2.70.194:50077 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 21:56:42 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:42 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:42.106235 11487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:42.101449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:58945)
0925 21:56:42.101618 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:42.101632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:42.101719 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:42.101943 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:42.101944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:42.102124 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:42.102172 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:42.102174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:42.102817 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:42.102818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:42.103363 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:42.103365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:42.103699 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:42.103700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:42.103752 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:42.104261 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:42.104265 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:42.106158 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
W20260925 21:56:42.106315 11485 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:42.101577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:58945 (local address 127.2.70.254:40965)
0925 21:56:42.101636 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:42.101638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:42.101704 (+ 66us) server_negotiation.cc:407] Connection header received
0925 21:56:42.101788 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:42.101789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:42.101865 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:42.101924 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:42.102249 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:42.102738 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:42.103444 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:42.103633 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:42.103696 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:42.106260 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:58945: BlockingRecv error: recv got EOF from 127.2.70.195:58945 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 21:56:42 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:42 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:42.859052 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:42.854067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:46059)
0925 21:56:42.854223 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:42.854237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:42.854312 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:42.854487 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:42.854488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:42.854654 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:42.854705 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:42.854706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:42.855515 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:42.855516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:42.856243 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:42.856246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:42.856644 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:42.856645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:42.856694 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:42.857164 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:42.857168 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:42.858988 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
W20260925 21:56:42.859229 11489 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:42.854189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:46059 (local address 127.2.70.254:40965)
0925 21:56:42.854325 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:42.854327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:42.854338 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:56:42.854364 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:42.854365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:42.854375 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:42.854438 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:42.854807 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:42.855430 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:42.856334 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:42.856559 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:42.856622 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:42.859140 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:46059: BlockingRecv error: recv got EOF from 127.2.70.193:46059 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
Sep 25 21:56:42 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:42 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:42.976500 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:42.971731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:33315)
0925 21:56:42.971888 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:42.971898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:42.972008 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:42.972137 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:42.972138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:42.972291 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:42.972340 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:42.972341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:42.973011 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:42.973012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:42.973606 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:42.973609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:42.973947 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:42.973948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:42.974000 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:42.974433 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:42.974437 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:42.976424 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
W20260925 21:56:42.976589 11489 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:42.971817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:33315 (local address 127.2.70.254:40965)
0925 21:56:42.971891 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:42.971894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:42.971992 (+ 98us) server_negotiation.cc:407] Connection header received
0925 21:56:42.972063 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:42.972064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:42.972074 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:42.972124 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:42.972389 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:42.972941 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:42.973682 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:42.973884 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:42.973944 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:42.976534 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:33315: BlockingRecv error: recv got EOF from 127.2.70.194:33315 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 21:56:43 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:43 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:43.111394 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:43.106815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:59533)
0925 21:56:43.106953 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:43.106963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:43.107026 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:43.107170 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:43.107171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:43.107339 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:43.107404 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:43.107406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:43.108109 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:43.108110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:43.108650 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:43.108652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:43.109000 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:43.109001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:43.109061 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:43.109539 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:43.109545 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:43.111297 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
W20260925 21:56:43.111464 11489 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:43.106839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:59533 (local address 127.2.70.254:40965)
0925 21:56:43.106960 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:43.106962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:43.107011 (+ 49us) server_negotiation.cc:407] Connection header received
0925 21:56:43.107088 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:43.107088 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:43.107101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:43.107153 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:43.107481 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:43.108043 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:43.108729 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:43.108924 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:43.108981 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:43.111408 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:59533: BlockingRecv error: recv got EOF from 127.2.70.195:59533 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 21:56:43 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:43 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:43.864426 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:43.859537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:39331)
0925 21:56:43.859683 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:43.859696 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:43.859751 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:43.859992 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:43.859993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:43.860162 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:43.860212 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:43.860213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:43.860964 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:43.860965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:43.861504 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:43.861506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:43.861935 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:43.861936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:43.861980 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:43.862416 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:43.862421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:43.864343 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
W20260925 21:56:43.864530 11493 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:43.859635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:39331 (local address 127.2.70.254:40965)
0925 21:56:43.859756 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:43.859758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:43.859793 (+ 35us) server_negotiation.cc:407] Connection header received
0925 21:56:43.859817 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:43.859818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:43.859860 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:43.859917 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:43.860318 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:43.860857 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:43.861597 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:43.861814 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:43.861887 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:43.864458 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:39331: BlockingRecv error: recv got EOF from 127.2.70.193:39331 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
Sep 25 21:56:43 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:43 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:43.981860 11494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:43.977062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:50737)
0925 21:56:43.977207 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:43.977219 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:43.977288 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:43.977440 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:43.977440 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:43.977600 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:43.977647 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:43.977649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:43.978438 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:43.978439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:43.978958 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:43.978960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:43.979341 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:43.979342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:43.979408 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:43.980001 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:43.980006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:43.981775 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
W20260925 21:56:43.981930 11493 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:43.977156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:50737 (local address 127.2.70.254:40965)
0925 21:56:43.977215 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:43.977218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:43.977284 (+ 66us) server_negotiation.cc:407] Connection header received
0925 21:56:43.977349 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:43.977350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:43.977362 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:43.977405 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:43.977720 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:43.978359 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:43.979035 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:43.979279 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:43.979340 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:43.981874 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:50737: BlockingRecv error: recv got EOF from 127.2.70.194:50737 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 21:56:44 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:44 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:44.116688 11495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:44.111958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:45159)
0925 21:56:44.112088 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:44.112099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:44.112159 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:44.112274 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:44.112275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:44.112432 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:44.112482 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:44.112484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:44.113457 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:44.113458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:44.114037 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:44.114040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:44.114384 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:44.114385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:44.114441 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:44.114865 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:44.114870 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:44.116610 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":93,"threads_started":1}
W20260925 21:56:44.116763 11493 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:44.112003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:45159 (local address 127.2.70.254:40965)
0925 21:56:44.112063 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:44.112065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:44.112141 (+ 76us) server_negotiation.cc:407] Connection header received
0925 21:56:44.112202 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:44.112203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:44.112214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:44.112267 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:44.112528 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:44.113374 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:44.114116 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:44.114313 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:44.114374 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:44.116710 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:45159: BlockingRecv error: recv got EOF from 127.2.70.195:45159 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 21:56:44 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:44 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:44.869482 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:44.864942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:42505)
0925 21:56:44.865091 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:44.865100 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:44.865154 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:44.865295 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:44.865296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:44.865464 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:44.865513 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:44.865515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:44.866146 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:44.866147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:44.866686 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:44.866688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:44.867082 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:44.867083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:44.867128 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:44.867586 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:44.867591 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:44.869418 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
W20260925 21:56:44.869668 11497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:44.865063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:42505 (local address 127.2.70.254:40965)
0925 21:56:44.865174 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:44.865176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:44.865183 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:56:44.865208 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:44.865209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:44.865218 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:44.865283 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:44.865571 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:44.866073 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:44.866796 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:44.867008 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:44.867069 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:44.869590 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:42505: BlockingRecv error: recv got EOF from 127.2.70.193:42505 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":52,"threads_started":1}
Sep 25 21:56:44 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:44 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:44.987074 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:44.982442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:48595)
0925 21:56:44.982565 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:44.982576 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:44.982642 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:44.982803 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:44.982804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:44.983033 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:44.983081 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:44.983083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:44.983741 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:44.983742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:44.984354 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:44.984357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:44.984749 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:44.984750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:44.984803 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:44.985294 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:44.985298 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:44.987004 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
W20260925 21:56:44.987197 11497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:44.982520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:48595 (local address 127.2.70.254:40965)
0925 21:56:44.982602 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:44.982604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:44.982629 (+ 25us) server_negotiation.cc:407] Connection header received
0925 21:56:44.982700 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:44.982700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:44.982711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:44.982756 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:44.983175 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:44.983672 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:44.984468 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:44.984665 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:44.984729 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:44.987121 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:48595: BlockingRecv error: recv got EOF from 127.2.70.194:48595 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 21:56:45 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:45 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:45.125871 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:45.117159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:37117)
0925 21:56:45.117317 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:45.117328 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:45.117395 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:45.117637 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:45.117638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:45.117920 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:45.117971 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:45.117973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:45.118658 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:45.118659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:45.119328 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:45.119330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:45.119652 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:45.119653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:45.119705 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:45.123548 (+ 3843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:45.123554 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:45.125794 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
W20260925 21:56:45.125977 11497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:45.117252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:37117 (local address 127.2.70.254:40965)
0925 21:56:45.117322 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:45.117324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:45.117379 (+ 55us) server_negotiation.cc:407] Connection header received
0925 21:56:45.117446 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:45.117447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:45.117459 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:45.117508 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:45.118063 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:45.118570 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:45.119406 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:45.119592 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:45.119645 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:45.125914 (+ 6269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:37117: BlockingRecv error: recv got EOF from 127.2.70.195:37117 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 21:56:45 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:45 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:45.875097 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:45.870078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:34349)
0925 21:56:45.870249 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:45.870262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:45.870347 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:45.870496 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:45.870497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:45.870710 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:45.870764 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:45.870766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:45.871542 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:45.871543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:45.872279 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:45.872282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:45.872660 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:45.872661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:45.872707 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:45.873225 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:45.873230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:45.875019 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
W20260925 21:56:45.875174 11501 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:45.870175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:34349 (local address 127.2.70.254:40965)
0925 21:56:45.870280 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:45.870281 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:45.870325 (+ 44us) server_negotiation.cc:407] Connection header received
0925 21:56:45.870404 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:45.870405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:45.870415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:45.870482 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:45.870884 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:45.871478 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:45.872383 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:45.872590 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:45.872652 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:45.875113 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:34349: BlockingRecv error: recv got EOF from 127.2.70.193:34349 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
Sep 25 21:56:45 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:45 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:45.992406 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:45.987573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:45125)
0925 21:56:45.987757 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:45.987769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:45.987865 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:45.988034 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:45.988035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:45.988193 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:45.988255 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:45.988256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:45.988964 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:45.988965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:45.989527 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:45.989529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:45.989884 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:45.989884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:45.989926 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:45.990386 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:45.990391 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:45.992328 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
W20260925 21:56:45.992491 11501 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:45.987696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:45125 (local address 127.2.70.254:40965)
0925 21:56:45.987751 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:45.987753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:45.987821 (+ 68us) server_negotiation.cc:407] Connection header received
0925 21:56:45.987929 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:45.987930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:45.987942 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:45.987991 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:45.988349 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:45.988868 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:45.989620 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:45.989819 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:45.989882 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:45.992431 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:45125: BlockingRecv error: recv got EOF from 127.2.70.194:45125 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 21:56:46 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:46 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:46.131091 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:46.126436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:49341)
0925 21:56:46.126618 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:46.126632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:46.126696 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:46.126858 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:46.126859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:46.127023 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:46.127072 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:46.127074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:46.127700 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:46.127701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:46.128389 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:46.128392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:46.128844 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:46.128844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:46.128904 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:46.129333 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:46.129337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:46.131021 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
W20260925 21:56:46.131206 11501 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:46.126625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:49341 (local address 127.2.70.254:40965)
0925 21:56:46.126675 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:46.126676 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:46.126685 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:46.126755 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:46.126756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:46.126767 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:46.126805 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:46.127137 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:46.127621 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:46.128502 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:46.128756 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:46.128819 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:46.131133 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:49341: BlockingRecv error: recv got EOF from 127.2.70.195:49341 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 21:56:46 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:46 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:46.880141 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:46.875578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:38637)
0925 21:56:46.875712 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:46.875723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:46.875781 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:46.875959 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:46.875960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:46.876131 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:46.876176 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:46.876178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:46.876809 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:46.876810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:46.877483 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:46.877486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:46.877810 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:46.877811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:46.877900 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:46.878342 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:46.878347 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:46.880052 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
W20260925 21:56:46.880219 11505 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:46.875652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:38637 (local address 127.2.70.254:40965)
0925 21:56:46.875765 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:46.875767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:46.875777 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:46.875852 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:46.875853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:46.875866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:46.875931 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:46.876232 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:46.876739 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:46.877557 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:46.877752 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:46.877804 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:46.880151 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:38637: BlockingRecv error: recv got EOF from 127.2.70.193:38637 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
Sep 25 21:56:46 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:46 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:46.997216 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:46.992903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:39891)
0925 21:56:46.993034 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:46.993044 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:46.993101 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:46.993219 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:46.993220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:46.993378 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:46.993422 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:46.993424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:46.994051 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:46.994052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:46.994574 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:46.994575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:46.994920 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:46.994921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:46.994966 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:46.995390 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:46.995394 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:46.997148 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
W20260925 21:56:46.997342 11505 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:46.992994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:39891 (local address 127.2.70.254:40965)
0925 21:56:46.993046 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:46.993048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:46.993096 (+ 48us) server_negotiation.cc:407] Connection header received
0925 21:56:46.993146 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:46.993147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:46.993158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:46.993208 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:46.993474 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:46.993985 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:46.994656 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:46.994853 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:46.994908 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:46.997271 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:39891: BlockingRecv error: recv got EOF from 127.2.70.194:39891 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 21:56:47 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:47 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:47.136036 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:47.131527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:36887)
0925 21:56:47.131641 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:47.131653 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:47.131730 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:47.131881 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:47.131882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:47.132059 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:47.132105 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:47.132107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:47.132754 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:47.132755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:47.133303 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:47.133304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:47.133716 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:47.133717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:47.133770 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:47.134232 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:47.134236 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:47.135965 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
W20260925 21:56:47.136201 11505 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:47.131585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:36887 (local address 127.2.70.254:40965)
0925 21:56:47.131651 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:47.131653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:47.131698 (+ 45us) server_negotiation.cc:407] Connection header received
0925 21:56:47.131781 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:47.131782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:47.131793 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:47.131825 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:47.132160 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:47.132678 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:47.133395 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:47.133625 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:47.133689 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:47.136134 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:36887: BlockingRecv error: recv got EOF from 127.2.70.195:36887 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 21:56:47 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:47 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:47.884826 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:47.880590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:48719)
0925 21:56:47.880751 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:47.880764 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:47.880822 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:47.880951 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:47.880952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:47.881116 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:47.881161 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:47.881162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:47.881832 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:47.881833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:47.882385 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:47.882386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:47.882716 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:47.882717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:47.882763 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:47.883166 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:47.883171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:47.884755 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
W20260925 21:56:47.884936 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:47.880669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:48719 (local address 127.2.70.254:40965)
0925 21:56:47.880796 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:47.880798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:47.880809 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:56:47.880867 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:47.880867 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:47.880881 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:47.880940 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:47.881245 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:47.881761 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:47.882454 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:47.882648 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:47.882700 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:47.884863 (+ 2163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:48719: BlockingRecv error: recv got EOF from 127.2.70.193:48719 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
Sep 25 21:56:47 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:47 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:48.002602 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:47.997746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:44123)
0925 21:56:47.997959 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:47.997972 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:47.998052 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:47.998180 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:47.998181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:47.998403 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:47.998462 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:47.998464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:47.999211 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:47.999212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:48.000041 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:48.000043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:48.000349 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:48.000350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:48.000386 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:48.000890 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:48.000895 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:48.002529 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":146,"threads_started":1}
W20260925 21:56:48.002718 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:47.997922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:44123 (local address 127.2.70.254:40965)
0925 21:56:47.997982 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:47.997984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:47.998032 (+ 48us) server_negotiation.cc:407] Connection header received
0925 21:56:47.998107 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:47.998108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:47.998121 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:47.998170 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:47.998537 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:47.999139 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:48.000109 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:48.000294 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:48.000346 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:48.002654 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:44123: BlockingRecv error: recv got EOF from 127.2.70.194:44123 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 21:56:48 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:48 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:48.140913 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:48.136528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:44693)
0925 21:56:48.136650 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:48.136661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:48.136713 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:48.136854 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:48.136855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:48.137022 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:48.137070 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:48.137072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:48.137722 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:48.137722 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:48.138251 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:48.138252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:48.138589 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:48.138590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:48.138626 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:48.139050 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:48.139054 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:48.140830 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
W20260925 21:56:48.140959 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:48.136611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:44693 (local address 127.2.70.254:40965)
0925 21:56:48.136684 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:48.136686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:48.136698 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:56:48.136769 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:48.136769 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:48.136784 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:48.136898 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:48.137148 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:48.137650 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:48.138336 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:48.138531 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:48.138587 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:48.140910 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:44693: BlockingRecv error: recv got EOF from 127.2.70.195:44693 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 21:56:48 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:48 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:48.889549 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:48.885325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:52637)
0925 21:56:48.885468 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:48.885478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:48.885545 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:48.885661 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:48.885662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:48.885818 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:48.885864 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:48.885865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:48.886464 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:48.886465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:48.887029 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:48.887030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:48.887384 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:48.887385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:48.887428 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:48.887887 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:48.887891 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:48.889471 (+ 1580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
W20260925 21:56:48.889695 11513 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:48.885414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:52637 (local address 127.2.70.254:40965)
0925 21:56:48.885506 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:48.885508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:48.885534 (+ 26us) server_negotiation.cc:407] Connection header received
0925 21:56:48.885592 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:48.885592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:48.885602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:48.885655 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:48.885920 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:48.886410 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:48.887117 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:48.887314 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:48.887382 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:48.889621 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:52637: BlockingRecv error: recv got EOF from 127.2.70.193:52637 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":45,"threads_started":1}
Sep 25 21:56:49 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:49 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:49.007939 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:49.003130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:42121)
0925 21:56:49.003300 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:49.003313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:49.003380 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:49.003523 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:49.003524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:49.003723 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:49.003786 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:49.003789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:49.004571 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:49.004572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:49.005142 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:49.005145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:49.005498 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:49.005499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:49.005556 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:49.006013 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:49.006017 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:49.007810 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
W20260925 21:56:49.008039 11513 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:49.003284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:42121 (local address 127.2.70.254:40965)
0925 21:56:49.003356 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:49.003357 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:49.003368 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:56:49.003448 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:49.003449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:49.003463 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:49.003518 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:49.003941 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:49.004481 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:49.005227 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:49.005429 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:49.005492 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:49.007970 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:42121: BlockingRecv error: recv got EOF from 127.2.70.194:42121 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 21:56:49 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:49 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:49.146009 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:49.141430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:45791)
0925 21:56:49.141572 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:49.141584 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:49.141664 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:49.141818 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:49.141819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:49.142029 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:49.142087 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:49.142090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:49.142753 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:49.142754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:49.143351 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:49.143354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:49.143693 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:49.143694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:49.143743 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:49.144226 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:49.144231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:49.145910 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
W20260925 21:56:49.146059 11513 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:49.141490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:45791 (local address 127.2.70.254:40965)
0925 21:56:49.141549 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:49.141551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:49.141644 (+ 93us) server_negotiation.cc:407] Connection header received
0925 21:56:49.141740 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:49.141742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:49.141757 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:49.141808 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:49.142156 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:49.142679 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:49.143419 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:49.143619 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:49.143690 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:49.146005 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:45791: BlockingRecv error: recv got EOF from 127.2.70.195:45791 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 21:56:49 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:49 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:49.894909 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:49.890090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:51737)
0925 21:56:49.890234 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:49.890245 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:49.890302 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:49.890427 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:49.890427 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:49.890604 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:49.890651 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:49.890653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:49.891443 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:49.891444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:49.892032 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:49.892035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:49.892396 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:49.892397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:49.892444 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:49.893004 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:49.893009 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:49.894833 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
W20260925 21:56:49.895031 11517 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:49.890177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:51737 (local address 127.2.70.254:40965)
0925 21:56:49.890315 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:49.890317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:49.890326 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:49.890349 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:49.890349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:49.890359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:49.890416 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:49.890756 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:49.891341 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:49.892112 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:49.892318 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:49.892373 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:49.894938 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:51737: BlockingRecv error: recv got EOF from 127.2.70.193:51737 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
Sep 25 21:56:50 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:50 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:50.012967 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:50.008430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:54459)
0925 21:56:50.008555 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:50.008567 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:50.008626 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:50.008784 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:50.008785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:50.008972 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:50.009018 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:50.009020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:50.009739 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:50.009740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:50.010269 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:50.010271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:50.010624 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:50.010625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:50.010678 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:50.011132 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:50.011136 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:50.012906 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
W20260925 21:56:50.013134 11517 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:50.008579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:54459 (local address 127.2.70.254:40965)
0925 21:56:50.008645 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:50.008647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:50.008658 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:56:50.008710 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:50.008711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:50.008721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:50.008764 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:50.009094 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:50.009658 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:50.010351 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:50.010563 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:50.010619 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:50.013017 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:54459: BlockingRecv error: recv got EOF from 127.2.70.194:54459 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 21:56:50 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:50 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.195@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:50.157053 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:50.146505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.195:60309)
0925 21:56:50.146648 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:50.146658 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:50.146714 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:50.146833 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:50.146833 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:50.147039 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:50.147085 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:50.147088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:50.147792 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:50.147794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:50.148435 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:50.148438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:50.148769 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:50.148770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:50.148863 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:50.154872 (+ 6009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:50.154878 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:50.156974 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
W20260925 21:56:50.157127 11517 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:50.146577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:60309 (local address 127.2.70.254:40965)
0925 21:56:50.146629 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:50.146630 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:50.146709 (+ 79us) server_negotiation.cc:407] Connection header received
0925 21:56:50.146760 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:50.146761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:50.146774 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:50.146822 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:50.147176 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:50.147701 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:50.148516 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:50.148704 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:50.148758 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:50.157074 (+ 8316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.195:60309: BlockingRecv error: recv got EOF from 127.2.70.195:60309 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 21:56:50 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:50 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.193@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:50.900417 11520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:50.895520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.193:58071)
0925 21:56:50.895650 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:50.895660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:50.895711 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:50.895909 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:50.895910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:50.896070 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:50.896119 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:50.896121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:50.896807 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:50.896808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:50.897611 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:50.897614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:50.898096 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:50.898097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:50.898156 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:50.898574 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:50.898579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:50.900357 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
W20260925 21:56:50.900525 11521 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:50.895623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:58071 (local address 127.2.70.254:40965)
0925 21:56:50.895757 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:50.895759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:50.895769 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:50.895794 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:50.895795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:50.895805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:50.895898 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:50.896197 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:50.896728 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:50.897695 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:50.898024 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:50.898097 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:50.900456 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.193:58071: BlockingRecv error: recv got EOF from 127.2.70.193:58071 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
Sep 25 21:56:51 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 21:56:51 dist-test-slave-2g0f krb5kdc[10463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.194@KRBTEST.COM for oryx/127.2.70.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 21:56:51.017817 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:51.013522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:40965 (local address 127.2.70.194:39449)
0925 21:56:51.013633 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:51.013645 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:51.013700 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:51.013817 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:51.013818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:51.013968 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:51.014017 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:51.014019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.014649 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.014650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:51.015167 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:51.015169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.015541 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.015542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:51.015578 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:51.016020 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:51.016025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:51.017746 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.254:40965: Server oryx/127.2.70.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
W20260925 21:56:51.017980 11521 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:51.013610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:39449 (local address 127.2.70.254:40965)
0925 21:56:51.013698 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:51.013700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:51.013710 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:51.013740 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:51.013741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:51.013752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:51.013801 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:51.014089 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.014587 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.015255 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.015466 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.015520 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:51.017903 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.194:39449: BlockingRecv error: recv got EOF from 127.2.70.194:39449 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 21:56:51.064949 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 11046
I20260925 21:56:51.068902 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 11198
I20260925 21:56:51.072678 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 11331
I20260925 21:56:51.076328 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 10976
2026-09-25T21:56:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17346 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T21:56:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:51Z Disabled control of system clock
I20260925 21:56:51.102702 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:41115
--webserver_interface=127.2.70.254
--webserver_port=0
--builtin_ntp_servers=127.2.70.212:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:41115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 21:56:51.188658 11526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:51.188875 11526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:51.188894 11526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:51.190172 11526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:51.190217 11526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:51.190229 11526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:51.190240 11526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:51.190253 11526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:51.191669 11526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:41115
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:41115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:51.191958 11526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:51.192260 11526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:51.197813 11532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:51.197845 11534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:51.197815 11531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:51.197861 11526 server_base.cc:1034] running on GCE node
I20260925 21:56:51.198274 11526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:51.198591 11526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:51.199749 11526 hybrid_clock.cc:648] HybridClock initialized: now 1790373411199734 us; error 31 us; skew 500 ppm
I20260925 21:56:51.201115 11526 webserver.cc:460] Webserver started at http://127.2.70.254:40095/ using document root <none> and password file <none>
I20260925 21:56:51.201362 11526 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:51.201411 11526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:51.201535 11526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:51.202601 11526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "c8f2bc2c69ef4a299eeb5bad7044b30d"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "d756927a373d2d7168c9c538729b1139"
server_key_iv: "c4d4e2880ed330d6c5b108d9c440eb0a"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.202957 11526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "c8f2bc2c69ef4a299eeb5bad7044b30d"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "d756927a373d2d7168c9c538729b1139"
server_key_iv: "c4d4e2880ed330d6c5b108d9c440eb0a"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.204303 11526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:51.205138 11540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.205295 11526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:51.205386 11526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "c8f2bc2c69ef4a299eeb5bad7044b30d"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "d756927a373d2d7168c9c538729b1139"
server_key_iv: "c4d4e2880ed330d6c5b108d9c440eb0a"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.205461 11526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:51.216293 11526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:51.216629 11526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:51.216766 11526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:51.220638 11526 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:41115
I20260925 21:56:51.220688 11592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:41115 every 8 connection(s)
I20260925 21:56:51.221031 11526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:51.221707 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:51.224520 11593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d: Bootstrap starting.
I20260925 21:56:51.225116 11593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:51.225389 11593 log.cc:824] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:51.226147 11593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d: No bootstrap required, opened a new log
I20260925 21:56:51.227212 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 11526
I20260925 21:56:51.227301 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:51.227530 2331 external_mini_cluster.cc:1428] Setting key fd7cb8501d17075b42e3ef1258b13b13
I20260925 21:56:51.228044 11593 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f2bc2c69ef4a299eeb5bad7044b30d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 41115 } }
I20260925 21:56:51.228235 11593 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:51.228271 11593 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f2bc2c69ef4a299eeb5bad7044b30d, State: Initialized, Role: FOLLOWER
I20260925 21:56:51.228397 11593 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f2bc2c69ef4a299eeb5bad7044b30d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 41115 } }
I20260925 21:56:51.228472 11593 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:51.228495 11593 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:51.228542 11593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:51.229084 11593 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f2bc2c69ef4a299eeb5bad7044b30d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 41115 } }
I20260925 21:56:51.229169 11593 leader_election.cc:302] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8f2bc2c69ef4a299eeb5bad7044b30d; no voters:
I20260925 21:56:51.229324 11593 leader_election.cc:288] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:51.229382 11598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:51.229601 11593 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:51.229612 11598 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [term 1 LEADER]: Becoming Leader. State: Replica: c8f2bc2c69ef4a299eeb5bad7044b30d, State: Running, Role: LEADER
I20260925 21:56:51.229756 11598 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f2bc2c69ef4a299eeb5bad7044b30d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 41115 } }
I20260925 21:56:51.230207 11599 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8f2bc2c69ef4a299eeb5bad7044b30d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f2bc2c69ef4a299eeb5bad7044b30d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 41115 } } }
I20260925 21:56:51.230293 11599 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:51.230510 11599 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f2bc2c69ef4a299eeb5bad7044b30d. Latest consensus state: current_term: 1 leader_uuid: "c8f2bc2c69ef4a299eeb5bad7044b30d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f2bc2c69ef4a299eeb5bad7044b30d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 41115 } } }
I20260925 21:56:51.230577 11599 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d [sys.catalog]: This master's current role is: LEADER
W20260925 21:56:51.231346 11614 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:56:51.231398 11614 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:56:51.231449 11607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:51.231632 11607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:51.233908 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:51.228153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.2.70.254:41115)
0925 21:56:51.228466 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:51.228477 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:51.228516 (+ 39us) server_negotiation.cc:407] Connection header received
0925 21:56:51.228650 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:51.228660 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:51.228716 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:51.228849 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:51.229075 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.230959 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.232096 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.232519 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.232637 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:51.233288 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:51.233292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:51.233292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:51.233477 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:51.233510 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:51.233521 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:51.233574 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:51.233658 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 21:56:51.233740 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
I20260925 21:56:51.236418 11607 catalog_manager.cc:1347] Generated new cluster ID: 3234a0383ea04cd3befca5a5cb677788
I20260925 21:56:51.236486 11607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:51.252573 11607 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:51.253386 11607 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:51.259819 11607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8f2bc2c69ef4a299eeb5bad7044b30d: Generated new TSK 0
I20260925 21:56:51.260077 11607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:56:51.269817 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:41115
--builtin_ntp_servers=127.2.70.212:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 21:56:51.362095 11617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:51.362316 11617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:51.362339 11617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:51.363559 11617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:51.363605 11617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:51.363631 11617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:51.365168 11617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:41115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:51.365417 11617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:51.365707 11617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:51.370628 11625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:51.370656 11623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:51.370628 11622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:51.370919 11617 server_base.cc:1034] running on GCE node
I20260925 21:56:51.371117 11617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:51.371358 11617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:51.372500 11617 hybrid_clock.cc:648] HybridClock initialized: now 1790373411372486 us; error 29 us; skew 500 ppm
I20260925 21:56:51.373746 11617 webserver.cc:460] Webserver started at http://127.2.70.193:41179/ using document root <none> and password file <none>
I20260925 21:56:51.373963 11617 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:51.374013 11617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:51.374140 11617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:51.375231 11617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "c58b9d7719e2499f89a860f12c54170e"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "5047c2dd6b27d379b7d97a5083ea0819"
server_key_iv: "c9437f9cd43f2789c32b377c5308a3dd"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.375593 11617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "c58b9d7719e2499f89a860f12c54170e"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "5047c2dd6b27d379b7d97a5083ea0819"
server_key_iv: "c9437f9cd43f2789c32b377c5308a3dd"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.376893 11617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:51.377648 11631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.377825 11617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:51.377907 11617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "c58b9d7719e2499f89a860f12c54170e"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "5047c2dd6b27d379b7d97a5083ea0819"
server_key_iv: "c9437f9cd43f2789c32b377c5308a3dd"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.377975 11617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:51.404946 11617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:51.405258 11617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:51.405400 11617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:51.405643 11617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:51.405975 11617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:51.406009 11617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.406038 11617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:51.406064 11617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.411782 11617 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:35915
I20260925 21:56:51.411928 11744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:35915 every 8 connection(s)
I20260925 21:56:51.412190 11617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:51.416018 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 11617
I20260925 21:56:51.416105 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:51.416312 2331 external_mini_cluster.cc:1428] Setting key 7a6de8f7410df9539df3507aa9c02233
I20260925 21:56:51.417408 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:41115
--builtin_ntp_servers=127.2.70.212:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 21:56:51.418121 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:51.412915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:38781 (local address 127.2.70.254:41115)
0925 21:56:51.412977 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:51.412978 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:51.413232 (+ 254us) server_negotiation.cc:407] Connection header received
0925 21:56:51.413781 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:51.413782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:51.413872 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:51.413928 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:51.414613 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.415135 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.416201 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.416546 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.416630 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:51.417822 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:51.417823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:51.417824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:51.417830 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:51.417850 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:51.417855 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:51.417912 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:51.417999 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 21:56:51.418039 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 21:56:51.418303 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:51.412902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:41115 (local address 127.2.70.193:38781)
0925 21:56:51.413111 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:51.413147 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:51.413664 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:51.414018 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:51.414022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:51.414031 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:51.414525 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:51.414529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.415229 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.415229 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:51.416115 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:51.416118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.416693 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.416695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:51.416881 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:51.417661 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:51.417662 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:51.417694 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:51.417721 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:51.417924 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:51.417929 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:51.417931 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:51.417988 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 21:56:51.418088 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260925 21:56:51.419152 11745 heartbeater.cc:347] Connected to a master server at 127.2.70.254:41115
I20260925 21:56:51.419252 11745 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:51.419476 11745 heartbeater.cc:511] Master 127.2.70.254:41115 requested a full tablet report, sending...
I20260925 21:56:51.420127 11557 ts_manager.cc:194] Registered new tserver with Master: c58b9d7719e2499f89a860f12c54170e (127.2.70.193:35915)
I20260925 21:56:51.420846 11557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.193:38781
W20260925 21:56:51.500294 11748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:51.500542 11748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:51.500564 11748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:51.501866 11748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:51.501917 11748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:51.501945 11748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:51.503340 11748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:41115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:51.503554 11748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:51.503813 11748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:56:51.508955 11748 server_base.cc:1034] running on GCE node
W20260925 21:56:51.508879 11753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:51.508987 11754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:51.508879 11756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:51.509389 11748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:51.509661 11748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:51.510842 11748 hybrid_clock.cc:648] HybridClock initialized: now 1790373411510793 us; error 62 us; skew 500 ppm
I20260925 21:56:51.512332 11748 webserver.cc:460] Webserver started at http://127.2.70.194:34393/ using document root <none> and password file <none>
I20260925 21:56:51.512578 11748 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:51.512629 11748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:51.512758 11748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:51.513845 11748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "f5808288090c4a0b97f61e48554e3730"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "8620a827764e51ad0ef3c47ab1d10a29"
server_key_iv: "650ccc7dd6b4211b6136c3a34dc12e1d"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.514189 11748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "f5808288090c4a0b97f61e48554e3730"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "8620a827764e51ad0ef3c47ab1d10a29"
server_key_iv: "650ccc7dd6b4211b6136c3a34dc12e1d"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.515512 11748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 21:56:51.516408 11762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.516599 11748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:56:51.516686 11748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "f5808288090c4a0b97f61e48554e3730"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "8620a827764e51ad0ef3c47ab1d10a29"
server_key_iv: "650ccc7dd6b4211b6136c3a34dc12e1d"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.516747 11748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:51.531458 11748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:51.531762 11748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:51.531952 11748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:51.532178 11748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:51.532511 11748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:51.532543 11748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.532574 11748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:51.532598 11748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.538162 11748 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:39989
I20260925 21:56:51.538218 11875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:39989 every 8 connection(s)
I20260925 21:56:51.538546 11748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:51.542848 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 11748
I20260925 21:56:51.542955 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:51.543125 2331 external_mini_cluster.cc:1428] Setting key ac0a820d5c647b8724d9ee509bfb2003
I20260925 21:56:51.544167 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:51.539475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:59365 (local address 127.2.70.254:41115)
0925 21:56:51.539569 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:51.539570 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:51.539815 (+ 245us) server_negotiation.cc:407] Connection header received
0925 21:56:51.540305 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:51.540307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:51.540316 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:51.540380 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:51.541048 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.541550 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.542536 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.542771 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.542843 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:51.543867 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:51.543868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:51.543869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:51.543873 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:51.543892 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:51.543897 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:51.543959 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:51.544057 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 21:56:51.544088 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 21:56:51.544687 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:41115
--builtin_ntp_servers=127.2.70.212:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 21:56:51.544791 11878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:51.539440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:41115 (local address 127.2.70.194:59365)
0925 21:56:51.539635 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:51.539678 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:51.540189 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:51.540448 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:51.540452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:51.540460 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:51.540966 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:51.540969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.541630 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.541631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:51.542460 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:51.542463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.542880 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.542881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:51.543046 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:51.543630 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:51.543630 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:51.543669 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:51.543698 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:51.543963 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:51.543969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:51.543973 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:51.544048 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 21:56:51.544153 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260925 21:56:51.545288 11876 heartbeater.cc:347] Connected to a master server at 127.2.70.254:41115
I20260925 21:56:51.545395 11876 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:51.545594 11876 heartbeater.cc:511] Master 127.2.70.254:41115 requested a full tablet report, sending...
I20260925 21:56:51.545974 11557 ts_manager.cc:194] Registered new tserver with Master: f5808288090c4a0b97f61e48554e3730 (127.2.70.194:39989)
I20260925 21:56:51.546550 11557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.194:59365
W20260925 21:56:51.629765 11879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:51.629987 11879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:51.630008 11879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:51.631179 11879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:51.631220 11879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:51.631243 11879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:51.632553 11879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:41115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:51.632750 11879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:51.633010 11879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:56:51.637894 11879 server_base.cc:1034] running on GCE node
W20260925 21:56:51.637847 11884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:51.637861 11885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:51.637852 11887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:51.638208 11879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:51.638447 11879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:51.639586 11879 hybrid_clock.cc:648] HybridClock initialized: now 1790373411639561 us; error 43 us; skew 500 ppm
I20260925 21:56:51.640830 11879 webserver.cc:460] Webserver started at http://127.2.70.195:44929/ using document root <none> and password file <none>
I20260925 21:56:51.641083 11879 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:51.641134 11879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:51.641263 11879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:51.642314 11879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "30a1312df0b94f59b00080aa209d20e0"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "f2ecb8fc12cdd3b00a6ab9ea327545e0"
server_key_iv: "4f28fca962f3069da693c9e783689bce"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.642656 11879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "30a1312df0b94f59b00080aa209d20e0"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "f2ecb8fc12cdd3b00a6ab9ea327545e0"
server_key_iv: "4f28fca962f3069da693c9e783689bce"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.643970 11879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:51.644655 11893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.644806 11879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:51.644888 11879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "30a1312df0b94f59b00080aa209d20e0"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "f2ecb8fc12cdd3b00a6ab9ea327545e0"
server_key_iv: "4f28fca962f3069da693c9e783689bce"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.644958 11879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:51.655589 11879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:51.655926 11879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:51.656071 11879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:51.656282 11879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:51.656590 11879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:51.656625 11879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.656654 11879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:51.656678 11879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.662093 11879 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:42827
I20260925 21:56:51.662151 12006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:42827 every 8 connection(s)
I20260925 21:56:51.662456 11879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:51.667443 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:51.663138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:55749 (local address 127.2.70.254:41115)
0925 21:56:51.663212 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:51.663215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:51.663398 (+ 183us) server_negotiation.cc:407] Connection header received
0925 21:56:51.663969 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:51.663970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:51.663981 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:51.664024 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:51.664659 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.665160 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.666160 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.666355 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.666408 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:51.667240 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:51.667241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:51.667241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:51.667246 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:51.667257 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:51.667261 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:51.667291 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:51.667357 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 21:56:51.667379 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 21:56:51.667627 12009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:51.663115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:41115 (local address 127.2.70.195:55749)
0925 21:56:51.663280 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:51.663315 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:51.663791 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:51.664084 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:51.664088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:51.664096 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:51.664596 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:51.664601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.665264 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.665264 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:51.666082 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:51.666085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:51.666453 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:51.666454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:51.666578 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:51.667120 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:51.667121 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:51.667152 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:51.667176 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:51.667301 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:51.667305 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:51.667307 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:51.667356 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 21:56:51.667438 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260925 21:56:51.668246 12007 heartbeater.cc:347] Connected to a master server at 127.2.70.254:41115
I20260925 21:56:51.668339 12007 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:51.668497 12007 heartbeater.cc:511] Master 127.2.70.254:41115 requested a full tablet report, sending...
I20260925 21:56:51.668928 11557 ts_manager.cc:194] Registered new tserver with Master: 30a1312df0b94f59b00080aa209d20e0 (127.2.70.195:42827)
I20260925 21:56:51.669296 11557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.195:55749
I20260925 21:56:51.669947 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 11879
I20260925 21:56:51.670068 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:51.670239 2331 external_mini_cluster.cc:1428] Setting key d8c692d638e7f99a204093c0185f6fca
I20260925 21:56:51.671090 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 21:56:51.672771 11597 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:51.672370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.2.70.254:41115)
0925 21:56:51.672417 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:51.672419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:51.672488 (+ 69us) server_negotiation.cc:407] Connection header received
0925 21:56:51.672557 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:51.672558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:51.672568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:51.672607 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:51.672714 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35238: BlockingRecv error: recv got EOF from 127.0.0.1:35238 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 21:56:51.672912 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 11617
I20260925 21:56:51.677253 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 11748
I20260925 21:56:51.681070 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 11879
I20260925 21:56:51.684966 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 11526
2026-09-25T21:56:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (607 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T21:56:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:51Z Disabled control of system clock
I20260925 21:56:51.709854 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:34881
--webserver_interface=127.2.70.254
--webserver_port=0
--builtin_ntp_servers=127.2.70.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:34881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 21:56:51.791661 12018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:51.791957 12018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:51.791985 12018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:51.793368 12018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:51.793431 12018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:51.793444 12018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:51.793457 12018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:51.793467 12018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:51.794974 12018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:34881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:34881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:51.795190 12018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:51.795452 12018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:51.800393 12023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:51.800390 12024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:51.800393 12026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:51.800468 12018 server_base.cc:1034] running on GCE node
I20260925 21:56:51.800938 12018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:51.801227 12018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:51.802392 12018 hybrid_clock.cc:648] HybridClock initialized: now 1790373411802379 us; error 32 us; skew 500 ppm
I20260925 21:56:51.803723 12018 webserver.cc:460] Webserver started at http://127.2.70.254:41613/ using document root <none> and password file <none>
I20260925 21:56:51.804039 12018 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:51.804091 12018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:51.804190 12018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:51.805218 12018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "1f1e3a18d63a45239aa65b3bdf1e5f7c"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "cc187e9f7fc61b549aaed0deca940aaf"
server_key_iv: "39b3dd95b7fc4d46217601866be6f28a"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.805518 12018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "1f1e3a18d63a45239aa65b3bdf1e5f7c"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "cc187e9f7fc61b549aaed0deca940aaf"
server_key_iv: "39b3dd95b7fc4d46217601866be6f28a"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.806740 12018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:51.807487 12032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.807641 12018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:51.807725 12018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "1f1e3a18d63a45239aa65b3bdf1e5f7c"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "cc187e9f7fc61b549aaed0deca940aaf"
server_key_iv: "39b3dd95b7fc4d46217601866be6f28a"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.807791 12018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:51.811278 12018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:51.811529 12018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:51.811630 12018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:51.815253 12018 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:34881
I20260925 21:56:51.815294 12084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:34881 every 8 connection(s)
I20260925 21:56:51.815603 12018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:51.816298 12085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:51.818996 12085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c: Bootstrap starting.
I20260925 21:56:51.819550 12085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:51.819808 12085 log.cc:824] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:51.820538 12085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c: No bootstrap required, opened a new log
I20260925 21:56:51.822144 12085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1e3a18d63a45239aa65b3bdf1e5f7c" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34881 } }
I20260925 21:56:51.822284 12085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:51.822321 12085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1e3a18d63a45239aa65b3bdf1e5f7c, State: Initialized, Role: FOLLOWER
I20260925 21:56:51.822458 12085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1e3a18d63a45239aa65b3bdf1e5f7c" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34881 } }
I20260925 21:56:51.822531 12085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:51.822561 12085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:51.822606 12085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:51.823137 12085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1e3a18d63a45239aa65b3bdf1e5f7c" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34881 } }
I20260925 21:56:51.823230 12085 leader_election.cc:302] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f1e3a18d63a45239aa65b3bdf1e5f7c; no voters:
I20260925 21:56:51.823381 12085 leader_election.cc:288] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:51.823444 12088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:51.823594 12088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [term 1 LEADER]: Becoming Leader. State: Replica: 1f1e3a18d63a45239aa65b3bdf1e5f7c, State: Running, Role: LEADER
I20260925 21:56:51.823624 12085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:51.823678 12088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1e3a18d63a45239aa65b3bdf1e5f7c" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34881 } }
I20260925 21:56:51.824203 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 12018
I20260925 21:56:51.824324 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:51.824286 12089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f1e3a18d63a45239aa65b3bdf1e5f7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1e3a18d63a45239aa65b3bdf1e5f7c" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34881 } } }
I20260925 21:56:51.824400 12089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:51.824532 2331 external_mini_cluster.cc:1428] Setting key e63254b555ec317eb084faf4e0be2085
I20260925 21:56:51.824862 12090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f1e3a18d63a45239aa65b3bdf1e5f7c. Latest consensus state: current_term: 1 leader_uuid: "1f1e3a18d63a45239aa65b3bdf1e5f7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1e3a18d63a45239aa65b3bdf1e5f7c" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34881 } } }
I20260925 21:56:51.824922 12090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c [sys.catalog]: This master's current role is: LEADER
W20260925 21:56:51.825508 12105 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:56:51.825591 12105 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:56:51.825675 12097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:51.825897 12097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:51.826783 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:51.825174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39536 (local address 127.2.70.254:34881)
0925 21:56:51.825377 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:51.825387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:51.825420 (+ 33us) server_negotiation.cc:407] Connection header received
0925 21:56:51.825544 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:51.825551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:51.825596 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:51.825735 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:51.826349 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:51.826353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:51.826353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:51.826357 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:51.826396 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:51.826410 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:51.826479 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:51.826555 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 21:56:51.826610 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260925 21:56:51.827804 12097 catalog_manager.cc:1347] Generated new cluster ID: a2932c1cb42b407e825a2531056825e1
I20260925 21:56:51.827883 12097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:56:51.850832 12097 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:51.851029 12097 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:51.854233 12097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f1e3a18d63a45239aa65b3bdf1e5f7c: Generated new TSK 0
I20260925 21:56:51.854441 12097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:56:51.862030 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34881
--builtin_ntp_servers=127.2.70.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 21:56:51.943271 12110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:51.943485 12110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:51.943516 12110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:51.944717 12110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:51.944761 12110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:51.944795 12110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:51.946135 12110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:51.946370 12110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:51.946648 12110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:51.952597 12116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:51.952739 12110 server_base.cc:1034] running on GCE node
W20260925 21:56:51.952616 12118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:51.952597 12115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:51.953116 12110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:51.953378 12110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:51.954540 12110 hybrid_clock.cc:648] HybridClock initialized: now 1790373411954526 us; error 27 us; skew 500 ppm
I20260925 21:56:51.955955 12110 webserver.cc:460] Webserver started at http://127.2.70.193:42661/ using document root <none> and password file <none>
I20260925 21:56:51.956197 12110 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:51.956251 12110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:51.956373 12110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:51.957736 12110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "7151ae0451624a8cb62b5f83c0862753"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "a018893374e35b87363c9491b986256a"
server_key_iv: "ef40da74ace90039648b183600ece9cb"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.958159 12110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "7151ae0451624a8cb62b5f83c0862753"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "a018893374e35b87363c9491b986256a"
server_key_iv: "ef40da74ace90039648b183600ece9cb"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.959496 12110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 21:56:51.960239 12124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.960387 12110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.960471 12110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "7151ae0451624a8cb62b5f83c0862753"
format_stamp: "Formatted at 2026-09-25 21:56:51 on dist-test-slave-2g0f"
server_key: "a018893374e35b87363c9491b986256a"
server_key_iv: "ef40da74ace90039648b183600ece9cb"
server_key_version: "encryptionkey@0"
I20260925 21:56:51.960541 12110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:51.963802 12110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:51.964097 12110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:51.964217 12110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:51.964411 12110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:51.964691 12110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:51.964723 12110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.964751 12110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:51.964767 12110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:51.970146 12110 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:38007
I20260925 21:56:51.970185 12237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:38007 every 8 connection(s)
I20260925 21:56:51.970484 12110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:51.973233 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:51.971187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:55059 (local address 127.2.70.254:34881)
0925 21:56:51.971244 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:51.971246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:51.971465 (+ 219us) server_negotiation.cc:407] Connection header received
0925 21:56:51.971937 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:51.971937 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:51.971946 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:51.971985 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:51.973069 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:51.973069 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:51.973070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:51.973072 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:51.973083 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:51.973086 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:51.973124 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:51.973171 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 21:56:51.973182 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 21:56:51.973415 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:51.971173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34881 (local address 127.2.70.193:55059)
0925 21:56:51.971338 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:51.971387 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:51.971821 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:51.972051 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:51.972055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:51.972063 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:51.972944 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:51.972945 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:51.972974 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:51.973003 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:51.973127 (+ 124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:51.973129 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:51.973131 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:51.973177 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 21:56:51.973223 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260925 21:56:51.973804 12238 heartbeater.cc:347] Connected to a master server at 127.2.70.254:34881
I20260925 21:56:51.973884 12238 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:51.973995 12238 heartbeater.cc:511] Master 127.2.70.254:34881 requested a full tablet report, sending...
I20260925 21:56:51.974423 12049 ts_manager.cc:194] Registered new tserver with Master: 7151ae0451624a8cb62b5f83c0862753 (127.2.70.193:38007)
I20260925 21:56:51.977139 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 12110
I20260925 21:56:51.977228 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:51.977399 2331 external_mini_cluster.cc:1428] Setting key 8a32a3195ec971ad1c16bebb93ac0f40
I20260925 21:56:51.978286 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34881
--builtin_ntp_servers=127.2.70.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 21:56:52.058656 12241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:52.058872 12241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:52.058902 12241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:52.060115 12241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:52.060158 12241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:52.060191 12241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:52.061568 12241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:52.061810 12241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:52.062083 12241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:52.066951 12246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:52.066959 12247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:52.067018 12249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:52.067131 12241 server_base.cc:1034] running on GCE node
I20260925 21:56:52.067349 12241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:52.067572 12241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:52.068711 12241 hybrid_clock.cc:648] HybridClock initialized: now 1790373412068701 us; error 25 us; skew 500 ppm
I20260925 21:56:52.069851 12241 webserver.cc:460] Webserver started at http://127.2.70.194:46155/ using document root <none> and password file <none>
I20260925 21:56:52.070076 12241 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:52.070125 12241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:52.070248 12241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:52.071314 12241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "e65315f1af7b4afb9cf8fe569bc06937"
format_stamp: "Formatted at 2026-09-25 21:56:52 on dist-test-slave-2g0f"
server_key: "fd8ea68e0f594447561a7c68aa890991"
server_key_iv: "6fd561a202ce1d9eebe5aa6973f50ab4"
server_key_version: "encryptionkey@0"
I20260925 21:56:52.071650 12241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "e65315f1af7b4afb9cf8fe569bc06937"
format_stamp: "Formatted at 2026-09-25 21:56:52 on dist-test-slave-2g0f"
server_key: "fd8ea68e0f594447561a7c68aa890991"
server_key_iv: "6fd561a202ce1d9eebe5aa6973f50ab4"
server_key_version: "encryptionkey@0"
I20260925 21:56:52.072904 12241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:52.073653 12255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:52.073822 12241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:52.073900 12241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "e65315f1af7b4afb9cf8fe569bc06937"
format_stamp: "Formatted at 2026-09-25 21:56:52 on dist-test-slave-2g0f"
server_key: "fd8ea68e0f594447561a7c68aa890991"
server_key_iv: "6fd561a202ce1d9eebe5aa6973f50ab4"
server_key_version: "encryptionkey@0"
I20260925 21:56:52.073968 12241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:52.077239 12241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:52.077471 12241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:52.077590 12241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:52.077765 12241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:52.078058 12241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:52.078089 12241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:52.078115 12241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:52.078138 12241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:52.083441 12241 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:45163
I20260925 21:56:52.083479 12368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:45163 every 8 connection(s)
I20260925 21:56:52.083794 12241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:52.086660 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:52.084497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:35953 (local address 127.2.70.254:34881)
0925 21:56:52.084551 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:52.084553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:52.084773 (+ 220us) server_negotiation.cc:407] Connection header received
0925 21:56:52.085268 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:52.085269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:52.085276 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:52.085335 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:52.086395 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:52.086396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:52.086397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:52.086401 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:52.086417 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:52.086421 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:52.086473 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:52.086576 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 21:56:52.086590 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 21:56:52.086829 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:52.084466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34881 (local address 127.2.70.194:35953)
0925 21:56:52.084656 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:52.084689 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:52.085163 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:52.085408 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:52.085412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:52.085420 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:52.086252 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:52.086252 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:52.086279 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:52.086307 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:52.086476 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:52.086481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:52.086484 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:52.086575 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 21:56:52.086638 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260925 21:56:52.087306 12369 heartbeater.cc:347] Connected to a master server at 127.2.70.254:34881
I20260925 21:56:52.087424 12369 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:52.087574 12369 heartbeater.cc:511] Master 127.2.70.254:34881 requested a full tablet report, sending...
I20260925 21:56:52.087954 12049 ts_manager.cc:194] Registered new tserver with Master: e65315f1af7b4afb9cf8fe569bc06937 (127.2.70.194:45163)
I20260925 21:56:52.092607 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 12241
I20260925 21:56:52.092687 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:52.092864 2331 external_mini_cluster.cc:1428] Setting key d7a48ca425736e6d7c30564280a323bb
I20260925 21:56:52.093729 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34881
--builtin_ntp_servers=127.2.70.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 21:56:52.175555 12372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:52.175770 12372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:52.175803 12372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:52.177042 12372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:52.177080 12372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:52.177114 12372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:52.178475 12372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:52.178700 12372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:52.178977 12372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:52.183921 12380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:52.183894 12378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:52.183900 12377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:52.184023 12372 server_base.cc:1034] running on GCE node
I20260925 21:56:52.184418 12372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:52.184636 12372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:52.185815 12372 hybrid_clock.cc:648] HybridClock initialized: now 1790373412185791 us; error 31 us; skew 500 ppm
I20260925 21:56:52.187053 12372 webserver.cc:460] Webserver started at http://127.2.70.195:39323/ using document root <none> and password file <none>
I20260925 21:56:52.187278 12372 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:52.187328 12372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:52.187453 12372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:52.188553 12372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "6b791814cdae4c38a1012ecc988f15ea"
format_stamp: "Formatted at 2026-09-25 21:56:52 on dist-test-slave-2g0f"
server_key: "154481211c1f8c68f5a16e0c142ed2a7"
server_key_iv: "2cc9b6e9c9072f313d0ccc9a1a2fb592"
server_key_version: "encryptionkey@0"
I20260925 21:56:52.188879 12372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "6b791814cdae4c38a1012ecc988f15ea"
format_stamp: "Formatted at 2026-09-25 21:56:52 on dist-test-slave-2g0f"
server_key: "154481211c1f8c68f5a16e0c142ed2a7"
server_key_iv: "2cc9b6e9c9072f313d0ccc9a1a2fb592"
server_key_version: "encryptionkey@0"
I20260925 21:56:52.190145 12372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:56:52.190785 12386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:52.190932 12372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:52.191010 12372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "6b791814cdae4c38a1012ecc988f15ea"
format_stamp: "Formatted at 2026-09-25 21:56:52 on dist-test-slave-2g0f"
server_key: "154481211c1f8c68f5a16e0c142ed2a7"
server_key_iv: "2cc9b6e9c9072f313d0ccc9a1a2fb592"
server_key_version: "encryptionkey@0"
I20260925 21:56:52.191066 12372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:52.194483 12372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:52.194729 12372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:52.194887 12372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:52.195065 12372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:52.195358 12372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:52.195392 12372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:52.195420 12372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:52.195442 12372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:52.200870 12372 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:43263
I20260925 21:56:52.200909 12499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:43263 every 8 connection(s)
I20260925 21:56:52.201208 12372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:52.204330 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:52.201895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:38981 (local address 127.2.70.254:34881)
0925 21:56:52.201961 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:52.201963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:52.202198 (+ 235us) server_negotiation.cc:407] Connection header received
0925 21:56:52.202851 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:52.202852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:52.202858 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:52.202927 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:52.204101 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:52.204102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:52.204103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:56:52.204107 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:52.204122 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:56:52.204126 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:52.204180 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:52.204258 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 21:56:52.204272 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 21:56:52.204560 12502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:52.201883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34881 (local address 127.2.70.195:38981)
0925 21:56:52.202075 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:52.202115 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:52.202693 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:52.203020 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:52.203024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:52.203032 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:52.203933 (+ 901us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:56:52.203934 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:52.203967 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:56:52.203997 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:52.204196 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:52.204199 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:52.204201 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:56:52.204255 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 21:56:52.204303 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260925 21:56:52.205051 12500 heartbeater.cc:347] Connected to a master server at 127.2.70.254:34881
I20260925 21:56:52.205138 12500 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:52.205274 12500 heartbeater.cc:511] Master 127.2.70.254:34881 requested a full tablet report, sending...
I20260925 21:56:52.205611 12049 ts_manager.cc:194] Registered new tserver with Master: 6b791814cdae4c38a1012ecc988f15ea (127.2.70.195:43263)
I20260925 21:56:52.208182 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 12372
I20260925 21:56:52.208264 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:52.208405 2331 external_mini_cluster.cc:1428] Setting key 3f6eab0b3635a642df8b44263e04f88d
I20260925 21:56:52.209347 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 21:56:52.211063 12107 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:52.210622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.2.70.254:34881)
0925 21:56:52.210678 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:52.210680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:52.210790 (+ 110us) server_negotiation.cc:407] Connection header received
0925 21:56:52.210818 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:52.210819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:52.210823 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:52.210871 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:52.211001 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39550: BlockingRecv error: recv got EOF from 127.0.0.1:39550 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 21:56:52.211148 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 12110
I20260925 21:56:52.215055 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 12241
I20260925 21:56:52.218904 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 12372
I20260925 21:56:52.222548 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 12018
2026-09-25T21:56:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (537 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:52 dist-test-slave-2g0f krb5kdc[12510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:52 dist-test-slave-2g0f krb5kdc[12510](info): set up 2 sockets
Sep 25 21:56:52 dist-test-slave-2g0f krb5kdc[12510](info): commencing operation
krb5kdc: starting...
W20260925 21:56:53.184546 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:53 dist-test-slave-2g0f krb5kdc[12510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:56:53.304203 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:32851
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:32851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:53.383715 12523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:53.383991 12523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:53.384029 12523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:53.385267 12523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:56:53.385322 12523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:53.385341 12523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:53.385358 12523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:56:53.385373 12523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:56:53.387073 12523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:32851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:32851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:53.387347 12523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:53.387586 12523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:53.392443 12529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:53.392580 12523 server_base.cc:1034] running on GCE node
W20260925 21:56:53.392582 12531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:53.392463 12528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:53.392966 12523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:53.393249 12523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:53.394438 12523 hybrid_clock.cc:648] HybridClock initialized: now 1790373413394419 us; error 33 us; skew 500 ppm
Sep 25 21:56:53 dist-test-slave-2g0f krb5kdc[12510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373413, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:53.397029 12523 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:56:53.397518 12523 webserver.cc:460] Webserver started at http://127.2.70.254:44007/ using document root <none> and password file <none>
I20260925 21:56:53.397733 12523 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:53.397782 12523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:53.397902 12523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:53.399060 12523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "61f0689985494e709fa5c5d08683f8b5"
format_stamp: "Formatted at 2026-09-25 21:56:53 on dist-test-slave-2g0f"
server_key: "f0a4441823a908bea22c6ddf1ed0ceda"
server_key_iv: "7f338da5fd63c8125a9747e3240dd7e7"
server_key_version: "encryptionkey@0"
I20260925 21:56:53.399528 12523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "61f0689985494e709fa5c5d08683f8b5"
format_stamp: "Formatted at 2026-09-25 21:56:53 on dist-test-slave-2g0f"
server_key: "f0a4441823a908bea22c6ddf1ed0ceda"
server_key_iv: "7f338da5fd63c8125a9747e3240dd7e7"
server_key_version: "encryptionkey@0"
I20260925 21:56:53.401055 12523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:53.401970 12538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:53.402140 12523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:53.402225 12523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "61f0689985494e709fa5c5d08683f8b5"
format_stamp: "Formatted at 2026-09-25 21:56:53 on dist-test-slave-2g0f"
server_key: "f0a4441823a908bea22c6ddf1ed0ceda"
server_key_iv: "7f338da5fd63c8125a9747e3240dd7e7"
server_key_version: "encryptionkey@0"
I20260925 21:56:53.402295 12523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:53.413662 12523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:53.415560 12523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:53.415693 12523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:53.419415 12523 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:32851
I20260925 21:56:53.419477 12590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:32851 every 8 connection(s)
I20260925 21:56:53.419785 12523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:56:53.420727 12591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:53.423010 12591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5: Bootstrap starting.
I20260925 21:56:53.423534 12591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:53.423754 12591 log.cc:824] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:53.424469 12591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5: No bootstrap required, opened a new log
I20260925 21:56:53.426082 12591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f0689985494e709fa5c5d08683f8b5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32851 } }
I20260925 21:56:53.426221 12591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:53.426244 12591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f0689985494e709fa5c5d08683f8b5, State: Initialized, Role: FOLLOWER
I20260925 21:56:53.426347 12591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f0689985494e709fa5c5d08683f8b5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32851 } }
I20260925 21:56:53.426402 12591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:56:53.426429 12591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:56:53.426469 12591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:53.427313 12591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f0689985494e709fa5c5d08683f8b5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32851 } }
I20260925 21:56:53.427412 12591 leader_election.cc:302] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61f0689985494e709fa5c5d08683f8b5; no voters:
I20260925 21:56:53.427561 12591 leader_election.cc:288] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:56:53.427603 12594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:53.427779 12594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [term 1 LEADER]: Becoming Leader. State: Replica: 61f0689985494e709fa5c5d08683f8b5, State: Running, Role: LEADER
I20260925 21:56:53.427819 12591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:56:53.427958 12594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f0689985494e709fa5c5d08683f8b5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32851 } }
I20260925 21:56:53.428313 12595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61f0689985494e709fa5c5d08683f8b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f0689985494e709fa5c5d08683f8b5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32851 } } }
I20260925 21:56:53.428412 12595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:53.428573 12596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61f0689985494e709fa5c5d08683f8b5. Latest consensus state: current_term: 1 leader_uuid: "61f0689985494e709fa5c5d08683f8b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f0689985494e709fa5c5d08683f8b5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32851 } } }
I20260925 21:56:53.428601 12603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:56:53.428635 12596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5 [sys.catalog]: This master's current role is: LEADER
I20260925 21:56:53.428977 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 12523
I20260925 21:56:53.429092 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:56:53.429191 12603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:56:53.429296 2331 external_mini_cluster.cc:1428] Setting key da8e6e3209832294880647f534fae4f0
I20260925 21:56:53.430967 12603 catalog_manager.cc:1347] Generated new cluster ID: d122ab72e2a248e5a9348c2978443529
I20260925 21:56:53.431021 12603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 21:56:53 dist-test-slave-2g0f krb5kdc[12510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:53.439039 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.429781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.2.70.254:32851)
0925 21:56:53.429946 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:53.429956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:53.430024 (+ 68us) server_negotiation.cc:407] Connection header received
0925 21:56:53.430186 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:53.430194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:53.430253 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:53.430393 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:53.431044 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.432044 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.432756 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.433185 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.433278 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.435445 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:53.435449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:53.435449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:53.435461 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:53.437540 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:53.437858 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:53.437861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:53.437862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:53.437961 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:53.438138 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:53.438139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:53.438139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:53.438381 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:53.438542 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:53.438615 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 21:56:53.438857 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260925 21:56:53.449059 12603 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:56:53.449790 12603 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:56:53.457777 12603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5: Generated new TSK 0
I20260925 21:56:53.457965 12603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:56:53.487457 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:32851
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:53.568476 12617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:53.568699 12617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:53.568723 12617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:53.569886 12617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:53.569929 12617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:53.569954 12617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:56:53.571419 12617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:53.571652 12617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:53.571897 12617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:53.576713 12623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:53.576733 12622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:53.576716 12625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:53.576853 12617 server_base.cc:1034] running on GCE node
I20260925 21:56:53.577263 12617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:53.577551 12617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:53.578758 12617 hybrid_clock.cc:648] HybridClock initialized: now 1790373413578743 us; error 33 us; skew 500 ppm
Sep 25 21:56:53 dist-test-slave-2g0f krb5kdc[12510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373413, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:53.581405 12617 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:56:53.581940 12617 webserver.cc:460] Webserver started at http://127.2.70.193:34117/ using document root <none> and password file <none>
I20260925 21:56:53.582165 12617 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:53.582212 12617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:53.582336 12617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:53.583422 12617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "e108befa9d694cbe808652c162ced9f2"
format_stamp: "Formatted at 2026-09-25 21:56:53 on dist-test-slave-2g0f"
server_key: "e1e04e0777e42e592e3c0f618ab9512c"
server_key_iv: "fb14672875b35dd325ab5c69a55cb664"
server_key_version: "encryptionkey@0"
I20260925 21:56:53.583756 12617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "e108befa9d694cbe808652c162ced9f2"
format_stamp: "Formatted at 2026-09-25 21:56:53 on dist-test-slave-2g0f"
server_key: "e1e04e0777e42e592e3c0f618ab9512c"
server_key_iv: "fb14672875b35dd325ab5c69a55cb664"
server_key_version: "encryptionkey@0"
I20260925 21:56:53.585090 12617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:56:53.585892 12632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:53.586062 12617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:56:53.586140 12617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "e108befa9d694cbe808652c162ced9f2"
format_stamp: "Formatted at 2026-09-25 21:56:53 on dist-test-slave-2g0f"
server_key: "e1e04e0777e42e592e3c0f618ab9512c"
server_key_iv: "fb14672875b35dd325ab5c69a55cb664"
server_key_version: "encryptionkey@0"
I20260925 21:56:53.586206 12617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:53.602674 12617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:53.605062 12617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:53.605222 12617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:53.605458 12617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:53.605759 12617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:53.605793 12617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:53.605822 12617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:53.605846 12617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:53.611491 12617 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:46855
I20260925 21:56:53.611558 12745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:46855 every 8 connection(s)
I20260925 21:56:53.611910 12617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:56:53.612273 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 12617
I20260925 21:56:53.612373 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:56:53.612569 2331 external_mini_cluster.cc:1428] Setting key cbca642d5dce04730416254ba0937b06
Sep 25 21:56:53 dist-test-slave-2g0f krb5kdc[12510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373413, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
I20260925 21:56:53.621346 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.612717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:45655 (local address 127.2.70.254:32851)
0925 21:56:53.612772 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:53.612774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:53.612929 (+ 155us) server_negotiation.cc:407] Connection header received
0925 21:56:53.613423 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:53.613424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:53.613438 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:53.613498 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:53.614517 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.615017 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.616092 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.616403 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.616474 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.618514 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:53.618516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:53.618517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:53.618525 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:53.619985 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:53.620434 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:53.620435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:53.620436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:53.620490 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:53.620762 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:53.620763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:53.620763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:53.620868 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:53.620920 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:53.621203 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 21:56:53.621277 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 21:56:53.621541 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.612662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32851 (local address 127.2.70.193:45655)
0925 21:56:53.612816 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:53.612859 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:53.613332 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:53.613561 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:53.613564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:53.614125 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:53.614403 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:53.614410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.615097 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.615098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:53.616015 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:53.616018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.616493 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.616494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:53.616595 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.617098 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:53.617106 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:53.618404 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:53.620088 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:53.620094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:53.620096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:53.620353 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:53.620581 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:53.620582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:53.620582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:53.620700 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:53.620948 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:53.620954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:53.621088 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 21:56:53.621182 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 21:56:53.621313 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260925 21:56:53.622040 12746 heartbeater.cc:347] Connected to a master server at 127.2.70.254:32851
I20260925 21:56:53.622143 12746 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:53.622318 12746 heartbeater.cc:511] Master 127.2.70.254:32851 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
I20260925 21:56:53.622869 12555 ts_manager.cc:194] Registered new tserver with Master: e108befa9d694cbe808652c162ced9f2 (127.2.70.193:46855)
I20260925 21:56:53.623579 12555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:45655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:56:53.639154 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:32851
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:53.720752 12751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:53.720990 12751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:53.721014 12751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:53.722172 12751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:53.722206 12751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:53.722230 12751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:56:53.723667 12751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:53.723902 12751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:53.724087 12751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:53.729306 12759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:53.729319 12757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:53.729291 12756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:53.729323 12751 server_base.cc:1034] running on GCE node
I20260925 21:56:53.729715 12751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:53.729969 12751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:53.731146 12751 hybrid_clock.cc:648] HybridClock initialized: now 1790373413731131 us; error 33 us; skew 500 ppm
Sep 25 21:56:53 dist-test-slave-2g0f krb5kdc[12510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373413, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:53.733687 12751 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:56:53.734218 12751 webserver.cc:460] Webserver started at http://127.2.70.194:40217/ using document root <none> and password file <none>
I20260925 21:56:53.734431 12751 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:53.734483 12751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:53.734614 12751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:53.735692 12751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "86d3ec225ed14f38bf66aca88d550d5c"
format_stamp: "Formatted at 2026-09-25 21:56:53 on dist-test-slave-2g0f"
server_key: "4a48557add5cf6e10242be8ebb7797f8"
server_key_iv: "1fcf85d424e83d394d47cd4ca2afe7b5"
server_key_version: "encryptionkey@0"
I20260925 21:56:53.736076 12751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "86d3ec225ed14f38bf66aca88d550d5c"
format_stamp: "Formatted at 2026-09-25 21:56:53 on dist-test-slave-2g0f"
server_key: "4a48557add5cf6e10242be8ebb7797f8"
server_key_iv: "1fcf85d424e83d394d47cd4ca2afe7b5"
server_key_version: "encryptionkey@0"
I20260925 21:56:53.737422 12751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:56:53.738125 12766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:53.738312 12751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:56:53.738405 12751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "86d3ec225ed14f38bf66aca88d550d5c"
format_stamp: "Formatted at 2026-09-25 21:56:53 on dist-test-slave-2g0f"
server_key: "4a48557add5cf6e10242be8ebb7797f8"
server_key_iv: "1fcf85d424e83d394d47cd4ca2afe7b5"
server_key_version: "encryptionkey@0"
I20260925 21:56:53.738462 12751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:53.772689 12751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:53.774816 12751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:53.774953 12751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:53.775170 12751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:53.775470 12751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:53.775502 12751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:53.775532 12751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:53.775555 12751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:53.780999 12751 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:44973
I20260925 21:56:53.781045 12879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:44973 every 8 connection(s)
I20260925 21:56:53.781354 12751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:56:53.784319 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 12751
I20260925 21:56:53.784425 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:56:53.784602 2331 external_mini_cluster.cc:1428] Setting key 60627f50f776dccb286894a4915dbdd2
Sep 25 21:56:53 dist-test-slave-2g0f krb5kdc[12510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373413, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:53.790886 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.782075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:60531 (local address 127.2.70.254:32851)
0925 21:56:53.782142 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:53.782145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:53.782336 (+ 191us) server_negotiation.cc:407] Connection header received
0925 21:56:53.782809 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:53.782810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:53.782821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:53.782872 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:53.783854 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.784356 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.785735 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.785932 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.785996 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.788086 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:53.788087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:53.788088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:53.788096 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:53.789560 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:53.789925 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:53.789926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:53.789927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:53.790000 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:53.790206 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:53.790207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:53.790207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:53.790348 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:53.790411 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:53.790696 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 21:56:53.790788 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 21:56:53.790964 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.782071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32851 (local address 127.2.70.194:60531)
0925 21:56:53.782230 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:53.782266 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:53.782732 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:53.782996 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:53.783000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:53.783550 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:53.783744 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:53.783748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.784463 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.784464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:53.785652 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:53.785655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.786006 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.786007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:53.786141 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.786663 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:53.786671 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:53.787994 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:53.789635 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:53.789641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:53.789642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:53.789855 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:53.790059 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:53.790060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:53.790060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:53.790147 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:53.790429 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:53.790435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:53.790571 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 21:56:53.790656 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 21:56:53.790779 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260925 21:56:53.791388 12880 heartbeater.cc:347] Connected to a master server at 127.2.70.254:32851
I20260925 21:56:53.791493 12880 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:53.791687 12880 heartbeater.cc:511] Master 127.2.70.254:32851 requested a full tablet report, sending...
I20260925 21:56:53.792152 12555 ts_manager.cc:194] Registered new tserver with Master: 86d3ec225ed14f38bf66aca88d550d5c (127.2.70.194:44973)
I20260925 21:56:53.792516 12555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:60531
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:56:53.810679 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:32851
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:56:53.891536 12885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:56:53.891753 12885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:56:53.891788 12885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:56:53.893004 12885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:56:53.893055 12885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:56:53.893092 12885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:56:53.894578 12885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:56:53.894829 12885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:56:53.895042 12885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:56:53.899884 12891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:53.899804 12893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:56:53.900025 12890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:56:53.900061 12885 server_base.cc:1034] running on GCE node
I20260925 21:56:53.900286 12885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:56:53.900527 12885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:56:53.901676 12885 hybrid_clock.cc:648] HybridClock initialized: now 1790373413901663 us; error 29 us; skew 500 ppm
Sep 25 21:56:53 dist-test-slave-2g0f krb5kdc[12510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373413, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:56:53.904018 12885 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:56:53.904421 12885 webserver.cc:460] Webserver started at http://127.2.70.195:45579/ using document root <none> and password file <none>
I20260925 21:56:53.904628 12885 fs_manager.cc:359] Metadata directory not provided
I20260925 21:56:53.904688 12885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:56:53.904817 12885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:56:53.905923 12885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "edacb81946df4bbf9ab872399c411a6f"
format_stamp: "Formatted at 2026-09-25 21:56:53 on dist-test-slave-2g0f"
server_key: "e5eab733e420801a44fd92210c6ce06f"
server_key_iv: "b34385b52478b6370b466fdad79c3b43"
server_key_version: "encryptionkey@0"
I20260925 21:56:53.906243 12885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "edacb81946df4bbf9ab872399c411a6f"
format_stamp: "Formatted at 2026-09-25 21:56:53 on dist-test-slave-2g0f"
server_key: "e5eab733e420801a44fd92210c6ce06f"
server_key_iv: "b34385b52478b6370b466fdad79c3b43"
server_key_version: "encryptionkey@0"
I20260925 21:56:53.907470 12885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:56:53.908227 12900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:53.908386 12885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:56:53.908468 12885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "edacb81946df4bbf9ab872399c411a6f"
format_stamp: "Formatted at 2026-09-25 21:56:53 on dist-test-slave-2g0f"
server_key: "e5eab733e420801a44fd92210c6ce06f"
server_key_iv: "b34385b52478b6370b466fdad79c3b43"
server_key_version: "encryptionkey@0"
I20260925 21:56:53.908524 12885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:56:53.934828 12885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:56:53.937037 12885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:56:53.937152 12885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:56:53.937340 12885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:56:53.937635 12885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:56:53.937673 12885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:53.937695 12885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:56:53.937709 12885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:56:53.943344 12885 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:35343
I20260925 21:56:53.943410 13013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:35343 every 8 connection(s)
I20260925 21:56:53.943730 12885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:56:53.945724 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 12885
I20260925 21:56:53.945830 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:56:53.946003 2331 external_mini_cluster.cc:1428] Setting key cfc09d19ce0aaa306ed7b80b2646ca45
Sep 25 21:56:53 dist-test-slave-2g0f krb5kdc[12510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373413, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:56:53.953079 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.944442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:41183 (local address 127.2.70.254:32851)
0925 21:56:53.944487 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:53.944489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:53.944653 (+ 164us) server_negotiation.cc:407] Connection header received
0925 21:56:53.945156 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:53.945157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:53.945168 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:53.945216 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:53.946306 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.946991 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.948088 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.948279 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.948328 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.950550 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:53.950551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:53.950552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:53.950559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:53.951755 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:53.952189 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:53.952189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:53.952190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:53.952242 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:53.952462 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:53.952463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:53.952463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:53.952578 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:53.952627 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:53.952942 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 21:56:53.953007 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 21:56:53.953274 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.944417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32851 (local address 127.2.70.195:41183)
0925 21:56:53.944552 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:53.944585 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:53.945071 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:53.945274 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:53.945277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:53.945960 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:56:53.946231 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:53.946237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.947098 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.947098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:53.948002 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:53.948005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.948366 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.948367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:53.948485 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.948961 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:56:53.948970 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:56:53.950440 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:56:53.951876 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:53.951884 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:53.951886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:53.952124 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:53.952311 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:56:53.952312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:56:53.952313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:56:53.952413 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:56:53.952656 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:56:53.952661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:56:53.952785 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 21:56:53.952923 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 21:56:53.953057 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260925 21:56:53.953698 13014 heartbeater.cc:347] Connected to a master server at 127.2.70.254:32851
I20260925 21:56:53.953788 13014 heartbeater.cc:464] Registering TS with master...
I20260925 21:56:53.953972 13014 heartbeater.cc:511] Master 127.2.70.254:32851 requested a full tablet report, sending...
I20260925 21:56:53.954324 12555 ts_manager.cc:194] Registered new tserver with Master: edacb81946df4bbf9ab872399c411a6f (127.2.70.195:35343)
I20260925 21:56:53.954697 12555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:41183
I20260925 21:56:53.958011 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:56:53.964895 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.959208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.2.70.254:32851)
0925 21:56:53.959276 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:53.959278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:53.959356 (+ 78us) server_negotiation.cc:407] Connection header received
0925 21:56:53.959397 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:53.959397 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:53.959407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:53.959462 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:56:53.960009 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.960503 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.961168 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.961363 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.961415 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.962580 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:56:53.962581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:56:53.962582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:56:53.962589 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:56:53.963966 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:53.964288 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:53.964288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:53.964289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:53.964339 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:56:53.964474 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:56:53.964475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:56:53.964475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:56:53.964619 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:56:53.964691 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:53.964764 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 21:56:53.964824 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 21:56:53.966334 12555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:56:53.968015 12555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:56:53.974277 13029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.970469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.2.70.195:35343)
0925 21:56:53.970643 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:53.970648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:53.970657 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:53.970698 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:53.970700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:53.970714 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:53.970778 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:53.970964 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.972155 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.973380 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.973892 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.973963 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.974022 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:53.974152 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 21:56:53.974201 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260925 21:56:53.974267 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.970200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:35343 (local address 127.0.0.1:33976)
0925 21:56:53.970354 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:53.970392 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:53.970455 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:53.970793 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:53.970796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:53.970798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:53.970864 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:53.970866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.972248 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.972249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:53.973290 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:53.973292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.973977 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.973978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:53.974053 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.974072 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 21:56:53.974133 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 21:56:53.974186 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":1,"thread_start_us":46,"threads_started":1}
I20260925 21:56:53.974591 13028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.970489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.2.70.194:44973)
0925 21:56:53.970925 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:53.970929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:53.970938 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:56:53.970974 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:53.970976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:53.970991 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:53.971062 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:53.971381 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.972388 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.973393 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.974108 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.974176 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.974230 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:53.974472 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 21:56:53.974517 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":80,"threads_started":1}
I20260925 21:56:53.974524 13026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.970224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:46855 (local address 127.0.0.1:51066)
0925 21:56:53.970562 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:53.970569 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:53.970606 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:53.971048 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:53.971049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:53.971052 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:53.971098 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:53.971100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.972006 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.972007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:53.973284 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:53.973287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.974265 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.974266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:53.974335 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.974339 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:53.974386 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 21:56:53.974427 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":6,"thread_start_us":90,"threads_started":1}
I20260925 21:56:53.974529 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.970224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:44973 (local address 127.0.0.1:36022)
0925 21:56:53.970358 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:53.970392 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:53.970455 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:53.971225 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:53.971225 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:53.971228 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:53.971264 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:53.971265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.972472 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.972472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:53.973304 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:53.973306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.974267 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.974268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:53.974344 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.974347 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:53.974386 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 21:56:53.974427 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260925 21:56:53.974854 13030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:53.970678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.2.70.193:46855)
0925 21:56:53.970818 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:53.970822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:53.970833 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:56:53.970868 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:53.970869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:53.970881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:53.970963 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:53.971156 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.971905 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.973368 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:53.974109 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:53.974176 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:53.974230 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:53.974599 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 21:56:53.974644 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260925 21:56:53.975375 12680 tablet_service.cc:1467] Processing CreateTablet for tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb (DEFAULT_TABLE table=test-table [id=722806599e974f4a833ead8b31ba3e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:53.975375 12948 tablet_service.cc:1467] Processing CreateTablet for tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb (DEFAULT_TABLE table=test-table [id=722806599e974f4a833ead8b31ba3e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:53.975481 12814 tablet_service.cc:1467] Processing CreateTablet for tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb (DEFAULT_TABLE table=test-table [id=722806599e974f4a833ead8b31ba3e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:56:53.975715 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:53.975715 12680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:53.975790 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:56:53.978340 13031 tablet_bootstrap.cc:492] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c: Bootstrap starting.
I20260925 21:56:53.978883 13032 tablet_bootstrap.cc:492] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2: Bootstrap starting.
I20260925 21:56:53.979022 13031 tablet_bootstrap.cc:654] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:53.979538 13032 tablet_bootstrap.cc:654] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:53.979547 13031 log.cc:824] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:53.980008 13033 tablet_bootstrap.cc:492] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f: Bootstrap starting.
I20260925 21:56:53.980152 13032 log.cc:824] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:53.980368 13031 tablet_bootstrap.cc:492] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c: No bootstrap required, opened a new log
I20260925 21:56:53.980475 13031 ts_tablet_manager.cc:1397] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:56:53.980616 13033 tablet_bootstrap.cc:654] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f: Neither blocks nor log segments found. Creating new log.
I20260925 21:56:53.981011 13032 tablet_bootstrap.cc:492] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2: No bootstrap required, opened a new log
I20260925 21:56:53.981081 13032 ts_tablet_manager.cc:1397] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:56:53.981110 13033 log.cc:824] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f: Log is configured to *not* fsync() on all Append() calls
I20260925 21:56:53.981876 13033 tablet_bootstrap.cc:492] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f: No bootstrap required, opened a new log
I20260925 21:56:53.981951 13033 ts_tablet_manager.cc:1397] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:56:53.982227 13031 raft_consensus.cc:348] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 } } peers { permanent_uuid: "86d3ec225ed14f38bf66aca88d550d5c" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44973 } } peers { permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 } }
I20260925 21:56:53.982368 13031 raft_consensus.cc:374] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:53.982400 13031 raft_consensus.cc:729] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d3ec225ed14f38bf66aca88d550d5c, State: Initialized, Role: FOLLOWER
I20260925 21:56:53.982525 13031 consensus_queue.cc:260] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 } } peers { permanent_uuid: "86d3ec225ed14f38bf66aca88d550d5c" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44973 } } peers { permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 } }
I20260925 21:56:53.982779 13031 ts_tablet_manager.cc:1428] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 21:56:53.982901 12880 heartbeater.cc:503] Master 127.2.70.254:32851 was elected leader, sending a full tablet report...
I20260925 21:56:53.982970 13032 raft_consensus.cc:348] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 } } peers { permanent_uuid: "86d3ec225ed14f38bf66aca88d550d5c" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44973 } } peers { permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 } }
I20260925 21:56:53.983115 13032 raft_consensus.cc:374] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:53.983155 13032 raft_consensus.cc:729] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e108befa9d694cbe808652c162ced9f2, State: Initialized, Role: FOLLOWER
I20260925 21:56:53.983258 13032 consensus_queue.cc:260] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 } } peers { permanent_uuid: "86d3ec225ed14f38bf66aca88d550d5c" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44973 } } peers { permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 } }
I20260925 21:56:53.983520 13032 ts_tablet_manager.cc:1428] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 21:56:53.983539 12746 heartbeater.cc:503] Master 127.2.70.254:32851 was elected leader, sending a full tablet report...
I20260925 21:56:53.984561 13033 raft_consensus.cc:348] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 } } peers { permanent_uuid: "86d3ec225ed14f38bf66aca88d550d5c" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44973 } } peers { permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 } }
I20260925 21:56:53.984678 13033 raft_consensus.cc:374] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:56:53.984696 13033 raft_consensus.cc:729] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edacb81946df4bbf9ab872399c411a6f, State: Initialized, Role: FOLLOWER
I20260925 21:56:53.984781 13033 consensus_queue.cc:260] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 } } peers { permanent_uuid: "86d3ec225ed14f38bf66aca88d550d5c" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44973 } } peers { permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 } }
I20260925 21:56:53.985050 13033 ts_tablet_manager.cc:1428] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 21:56:53.985096 13014 heartbeater.cc:503] Master 127.2.70.254:32851 was elected leader, sending a full tablet report...
W20260925 21:56:54.031948 12881 tablet.cc:2369] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 21:56:54.112684 12747 tablet.cc:2369] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 21:56:54.194329 13015 tablet.cc:2369] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:56:54.217325 13037 raft_consensus.cc:482] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:54.217458 13037 raft_consensus.cc:504] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 } } peers { permanent_uuid: "86d3ec225ed14f38bf66aca88d550d5c" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44973 } } peers { permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 } }
I20260925 21:56:54.217831 13037 leader_election.cc:288] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edacb81946df4bbf9ab872399c411a6f (127.2.70.195:35343), e108befa9d694cbe808652c162ced9f2 (127.2.70.193:46855)
I20260925 21:56:54.218241 13039 raft_consensus.cc:482] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:56:54.218430 13039 raft_consensus.cc:504] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 } } peers { permanent_uuid: "86d3ec225ed14f38bf66aca88d550d5c" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44973 } } peers { permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 } }
I20260925 21:56:54.219458 13039 leader_election.cc:288] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86d3ec225ed14f38bf66aca88d550d5c (127.2.70.194:44973), e108befa9d694cbe808652c162ced9f2 (127.2.70.193:46855)
I20260925 21:56:54.221401 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:54.218028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:35343 (local address 127.2.70.194:56639)
0925 21:56:54.218080 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:54.218094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:54.218162 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:54.218336 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:54.218336 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:54.218338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:54.218419 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:54.218422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.219324 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.219325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:54.220482 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:54.220485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.221167 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.221168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:54.221247 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:54.221250 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:54.221289 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 21:56:54.221327 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 21:56:54.221722 13029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:54.218167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:56639 (local address 127.2.70.195:35343)
0925 21:56:54.218222 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:54.218224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:54.218234 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:56:54.218263 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:54.218263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:54.218273 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:54.218330 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:54.218492 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.219145 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.220564 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.221071 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.221131 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:54.221140 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:54.221627 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 21:56:54.221663 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":31}
I20260925 21:56:54.221797 13030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:54.218114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:51567 (local address 127.2.70.193:46855)
0925 21:56:54.218155 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:54.218157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:54.218285 (+ 128us) server_negotiation.cc:407] Connection header received
0925 21:56:54.218343 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:54.218344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:54.218356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:54.218382 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:54.218554 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.219108 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.220907 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.221453 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.221497 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:54.221504 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:54.221699 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 21:56:54.221732 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 21:56:54.221946 12968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdf042ad1e94a45bcbf6cb5d0ecffbb" candidate_uuid: "86d3ec225ed14f38bf66aca88d550d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edacb81946df4bbf9ab872399c411a6f" is_pre_election: true
I20260925 21:56:54.222018 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:54.218028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:46855 (local address 127.2.70.194:51567)
0925 21:56:54.218226 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:54.218235 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:54.218281 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:54.218438 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:54.218439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:54.218440 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:54.218491 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:54.218493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.219190 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.219191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:54.220816 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:54.220818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.221557 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.221558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:54.221628 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:54.221631 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:56:54.221899 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 21:56:54.221939 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":500,"thread_start_us":89,"threads_started":1}
I20260925 21:56:54.222034 12968 raft_consensus.cc:2454] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86d3ec225ed14f38bf66aca88d550d5c in term 0.
I20260925 21:56:54.222220 12769 leader_election.cc:302] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86d3ec225ed14f38bf66aca88d550d5c, edacb81946df4bbf9ab872399c411a6f; no voters:
I20260925 21:56:54.222285 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:54.218976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:44973 (local address 127.2.70.195:56937)
0925 21:56:54.219124 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:54.219132 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:54.219173 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:54.219334 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:54.219335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:54.219336 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:54.219382 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:54.219383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.220253 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.220254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:54.221339 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:54.221341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.222080 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.222081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:54.222149 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:54.222153 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:56:54.222196 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 21:56:54.222230 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 21:56:54.222337 13037 raft_consensus.cc:2781] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:54.222374 13037 raft_consensus.cc:482] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:54.222394 13037 raft_consensus.cc:3037] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:54.222383 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:54.219011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:46855 (local address 127.2.70.195:45053)
0925 21:56:54.219211 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:54.219221 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:56:54.219262 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:56:54.219457 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:56:54.219458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:56:54.219459 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:56:54.219525 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:54.219526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.220322 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.220323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:54.221455 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:56:54.221458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.222162 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.222163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:56:54.222225 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:54.222230 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:56:54.222261 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 21:56:54.222301 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260925 21:56:54.222436 12700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdf042ad1e94a45bcbf6cb5d0ecffbb" candidate_uuid: "86d3ec225ed14f38bf66aca88d550d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e108befa9d694cbe808652c162ced9f2" is_pre_election: true
I20260925 21:56:54.222565 12700 raft_consensus.cc:2454] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86d3ec225ed14f38bf66aca88d550d5c in term 0.
I20260925 21:56:54.222831 13028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:54.219080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:56937 (local address 127.2.70.194:44973)
0925 21:56:54.219114 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:54.219116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:54.219224 (+ 108us) server_negotiation.cc:407] Connection header received
0925 21:56:54.219246 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:54.219247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:54.219256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:54.219298 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:54.219442 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.220151 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.221412 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.221885 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.221954 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:54.221961 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:54.222742 (+ 781us) server_negotiation.cc:299] Negotiation successful
0925 21:56:54.222780 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 21:56:54.222847 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:54.219242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:45053 (local address 127.2.70.193:46855)
0925 21:56:54.219348 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:54.219350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:54.219355 (+ 5us) server_negotiation.cc:407] Connection header received
0925 21:56:54.219373 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:54.219374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:54.219383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:54.219584 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:56:54.219623 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.220233 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.221539 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.222070 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.222480 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:54.222487 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:54.222503 (+ 16us) server_negotiation.cc:299] Negotiation successful
0925 21:56:54.222537 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
I20260925 21:56:54.223034 12700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdf042ad1e94a45bcbf6cb5d0ecffbb" candidate_uuid: "edacb81946df4bbf9ab872399c411a6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e108befa9d694cbe808652c162ced9f2" is_pre_election: true
I20260925 21:56:54.223058 12834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdf042ad1e94a45bcbf6cb5d0ecffbb" candidate_uuid: "edacb81946df4bbf9ab872399c411a6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d3ec225ed14f38bf66aca88d550d5c" is_pre_election: true
I20260925 21:56:54.223116 12700 raft_consensus.cc:2454] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edacb81946df4bbf9ab872399c411a6f in term 0.
I20260925 21:56:54.223287 13037 raft_consensus.cc:504] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 } } peers { permanent_uuid: "86d3ec225ed14f38bf66aca88d550d5c" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44973 } } peers { permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 } }
I20260925 21:56:54.223290 12902 leader_election.cc:302] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e108befa9d694cbe808652c162ced9f2, edacb81946df4bbf9ab872399c411a6f; no voters:
I20260925 21:56:54.223407 13039 raft_consensus.cc:2781] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:56:54.223450 13039 raft_consensus.cc:482] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:56:54.223438 12834 raft_consensus.cc:2379] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate edacb81946df4bbf9ab872399c411a6f in current term 1: Already voted for candidate 86d3ec225ed14f38bf66aca88d550d5c in this term.
I20260925 21:56:54.223472 13039 raft_consensus.cc:3037] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:54.223438 13037 leader_election.cc:288] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [CANDIDATE]: Term 1 election: Requested vote from peers edacb81946df4bbf9ab872399c411a6f (127.2.70.195:35343), e108befa9d694cbe808652c162ced9f2 (127.2.70.193:46855)
I20260925 21:56:54.223559 12968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdf042ad1e94a45bcbf6cb5d0ecffbb" candidate_uuid: "86d3ec225ed14f38bf66aca88d550d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edacb81946df4bbf9ab872399c411a6f"
I20260925 21:56:54.223682 12700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdf042ad1e94a45bcbf6cb5d0ecffbb" candidate_uuid: "86d3ec225ed14f38bf66aca88d550d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e108befa9d694cbe808652c162ced9f2"
I20260925 21:56:54.223739 12700 raft_consensus.cc:3037] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:56:54.224200 13039 raft_consensus.cc:504] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 } } peers { permanent_uuid: "86d3ec225ed14f38bf66aca88d550d5c" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44973 } } peers { permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 } }
I20260925 21:56:54.224341 12968 raft_consensus.cc:2379] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 86d3ec225ed14f38bf66aca88d550d5c in current term 1: Already voted for candidate edacb81946df4bbf9ab872399c411a6f in this term.
I20260925 21:56:54.224419 12700 raft_consensus.cc:2454] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86d3ec225ed14f38bf66aca88d550d5c in term 1.
I20260925 21:56:54.224442 13039 leader_election.cc:288] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [CANDIDATE]: Term 1 election: Requested vote from peers 86d3ec225ed14f38bf66aca88d550d5c (127.2.70.194:44973), e108befa9d694cbe808652c162ced9f2 (127.2.70.193:46855)
I20260925 21:56:54.224557 12768 leader_election.cc:302] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86d3ec225ed14f38bf66aca88d550d5c, e108befa9d694cbe808652c162ced9f2; no voters:
I20260925 21:56:54.224532 12834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdf042ad1e94a45bcbf6cb5d0ecffbb" candidate_uuid: "edacb81946df4bbf9ab872399c411a6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d3ec225ed14f38bf66aca88d550d5c"
I20260925 21:56:54.224630 12834 raft_consensus.cc:2379] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate edacb81946df4bbf9ab872399c411a6f in current term 1: Already voted for candidate 86d3ec225ed14f38bf66aca88d550d5c in this term.
I20260925 21:56:54.224675 13037 raft_consensus.cc:2781] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:56:54.224689 12700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdf042ad1e94a45bcbf6cb5d0ecffbb" candidate_uuid: "edacb81946df4bbf9ab872399c411a6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e108befa9d694cbe808652c162ced9f2"
I20260925 21:56:54.224761 12700 raft_consensus.cc:2379] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate edacb81946df4bbf9ab872399c411a6f in current term 1: Already voted for candidate 86d3ec225ed14f38bf66aca88d550d5c in this term.
I20260925 21:56:54.224830 13037 raft_consensus.cc:686] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 1 LEADER]: Becoming Leader. State: Replica: 86d3ec225ed14f38bf66aca88d550d5c, State: Running, Role: LEADER
I20260925 21:56:54.224929 13037 consensus_queue.cc:237] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 } } peers { permanent_uuid: "86d3ec225ed14f38bf66aca88d550d5c" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44973 } } peers { permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 } }
I20260925 21:56:54.224958 12902 leader_election.cc:302] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edacb81946df4bbf9ab872399c411a6f; no voters: 86d3ec225ed14f38bf66aca88d550d5c, e108befa9d694cbe808652c162ced9f2
I20260925 21:56:54.225044 13039 raft_consensus.cc:2726] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 21:56:54.225706 12555 catalog_manager.cc:5495] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 86d3ec225ed14f38bf66aca88d550d5c (127.2.70.194). New cstate: current_term: 1 leader_uuid: "86d3ec225ed14f38bf66aca88d550d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86d3ec225ed14f38bf66aca88d550d5c" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 } health_report { overall_health: UNKNOWN } } }
I20260925 21:56:54.256877 13028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:56:54.254493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.2.70.194:44973)
0925 21:56:54.254551 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 21:56:54.254552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:56:54.254560 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:56:54.254594 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:56:54.254595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:56:54.254616 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:56:54.254675 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:56:54.254801 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.255301 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.256109 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:56:54.256313 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:56:54.256367 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:56:54.256534 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:56:54.256638 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:56:54.256700 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 21:56:54.256781 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 21:56:54.256800 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 21:56:54.258661 12968 raft_consensus.cc:1261] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 1 FOLLOWER]: Refusing update from remote peer 86d3ec225ed14f38bf66aca88d550d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:54.258662 12700 raft_consensus.cc:1261] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2 [term 1 FOLLOWER]: Refusing update from remote peer 86d3ec225ed14f38bf66aca88d550d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:56:54.258937 13043 consensus_queue.cc:1035] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "edacb81946df4bbf9ab872399c411a6f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:54.259069 13037 consensus_queue.cc:1035] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e108befa9d694cbe808652c162ced9f2" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:56:54.261358 13046 mvcc.cc:205] Tried to move back new op lower bound from 7333369504800919552 to 7333369504665841664. Current Snapshot: MvccSnapshot[applied={T|T < 7333369504800919552}]
I20260925 21:56:54.261528 13047 mvcc.cc:205] Tried to move back new op lower bound from 7333369504800919552 to 7333369504665841664. Current Snapshot: MvccSnapshot[applied={T|T < 7333369504800919552}]
I20260925 21:56:54.262446 13048 mvcc.cc:205] Tried to move back new op lower bound from 7333369504800919552 to 7333369504665841664. Current Snapshot: MvccSnapshot[applied={T|T < 7333369504800919552}]
I20260925 21:56:54.263800 12555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:54.263917 12555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:56:54.264674 12555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61f0689985494e709fa5c5d08683f8b5: Sending DeleteTablet for 3 replicas of tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb
I20260925 21:56:54.265009 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:54 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:33976
I20260925 21:56:54.265045 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:54 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:36022
I20260925 21:56:54.265275 12680 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:56:54 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:51066
I20260925 21:56:54.265470 13055 tablet_replica.cc:330] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f: stopping tablet replica
I20260925 21:56:54.265498 13057 tablet_replica.cc:330] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2: stopping tablet replica
I20260925 21:56:54.265592 13057 raft_consensus.cc:2229] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P e108befa9d694cbe808652c162ced9f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:54.265625 13055 raft_consensus.cc:2229] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:56:54.265733 13055 raft_consensus.cc:2258] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:54.265862 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 12617
I20260925 21:56:54.265931 13056 tablet_replica.cc:330] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c: stopping tablet replica
I20260925 21:56:54.266019 13056 raft_consensus.cc:2229] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:56:54.266094 13056 raft_consensus.cc:2258] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:56:54.266793 13055 ts_tablet_manager.cc:1905] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:54.267146 13056 ts_tablet_manager.cc:1905] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:56:54.268798 13056 ts_tablet_manager.cc:1918] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:54.268872 13056 log.cc:1196] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/dcdf042ad1e94a45bcbf6cb5d0ecffbb
I20260925 21:56:54.269101 13056 ts_tablet_manager.cc:1939] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P 86d3ec225ed14f38bf66aca88d550d5c: Deleting consensus metadata
I20260925 21:56:54.269555 12542 catalog_manager.cc:4841] TS 86d3ec225ed14f38bf66aca88d550d5c (127.2.70.194:44973): tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb (table test-table [id=722806599e974f4a833ead8b31ba3e71]) successfully deleted
I20260925 21:56:54.271978 13055 ts_tablet_manager.cc:1918] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:56:54.272066 13055 log.cc:1196] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/dcdf042ad1e94a45bcbf6cb5d0ecffbb
I20260925 21:56:54.272357 13055 ts_tablet_manager.cc:1939] T dcdf042ad1e94a45bcbf6cb5d0ecffbb P edacb81946df4bbf9ab872399c411a6f: Deleting consensus metadata
W20260925 21:56:54.272436 12540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:46855 (error 108)
I20260925 21:56:54.272800 12541 catalog_manager.cc:4841] TS edacb81946df4bbf9ab872399c411a6f (127.2.70.195:35343): tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb (table test-table [id=722806599e974f4a833ead8b31ba3e71]) successfully deleted
W20260925 21:56:54.272979 13026 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:56:54.272847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:46855 (local address 127.0.0.1:51068)
0925 21:56:54.272901 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 21:56:54.272935 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:46855: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 21:56:54.273100 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 12751
W20260925 21:56:54.273094 12540 catalog_manager.cc:4584] TS e108befa9d694cbe808652c162ced9f2 (127.2.70.193:46855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dcdf042ad1e94a45bcbf6cb5d0ecffbb: Network error: Client connection negotiation failed: client connection to 127.2.70.193:46855: connect: Connection reset by peer (error 104)
I20260925 21:56:54.277329 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 12885
I20260925 21:56:54.281173 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 12523
2026-09-25T21:56:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2059 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:56:54 dist-test-slave-2g0f krb5kdc[13061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:56:54 dist-test-slave-2g0f krb5kdc[13061](info): set up 2 sockets
Sep 25 21:56:54 dist-test-slave-2g0f krb5kdc[13061](info): commencing operation
krb5kdc: starting...
W20260925 21:56:55.244653 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:56:55 dist-test-slave-2g0f krb5kdc[13061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:56:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:56:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.70.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.70.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/rangeradmin_127.2.70.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/rangeradmin_127.2.70.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.70.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.70.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/rangerlookup_127.2.70.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/rangerlookup_127.2.70.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.70.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.70.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/HTTP_127.2.70.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/HTTP_127.2.70.212@KRBTEST.COM.keytab.
I20260925 21:56:55.411268 2331 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdR78Ha/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/postgres -l logfile start
2026-09-25 21:56:57.940 UTC [13091] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 21:56:57.940 UTC [13091] LOG: listening on IPv4 address "127.2.70.212", port 47233
2026-09-25 21:56:57.944 UTC [13091] LOG: listening on Unix socket "/tmp/.s.PGSQL.47233"
2026-09-25 21:56:57.972 UTC [13094] LOG: database system was shut down at 2026-09-25 21:56:56 UTC
2026-09-25 21:56:57.976 UTC [13091] LOG: database system is ready to accept connections
I20260925 21:56:58.836376 2331 mini_postgres.cc:96] Postgres bound to 47233
2026-09-25 21:56:58.842 UTC [13102] FATAL: database "slave" does not exist
127.2.70.212:47233 - accepting connections
I20260925 21:56:58.842698 2331 mini_ranger.cc:161] Starting Ranger...
I20260925 21:56:58.857398 2331 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 21:56:59.347999 2331 mini_ranger.cc:87] Created ranger Postgres database
I20260925 21:56:59.348064 2331 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-admin
2026-09-25 21:56:59,614 [I] DB FLAVOR :POSTGRES
2026-09-25 21:56:59,614 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 21:56:59,614 [I] Checking connection..
2026-09-25 21:56:59,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 21:56:59,855 [I] Checking connection passed.
2026-09-25 21:56:59,855 [I] --------- Verifying version history table ---------
2026-09-25 21:56:59,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 21:57:00,117 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 21:57:00,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 21:57:00,348 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 21:57:00,349 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 21:57:00,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 21:57:00.588 UTC [13166] WARNING: there is no transaction in progress
2026-09-25 21:57:00,594 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 21:57:00,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 21:57:00,854 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 21:57:00,854 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 21:57:00,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 21:57:01,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 21:57:01,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2g0f.c.gcp-upstream.internal','N') ;"
2026-09-25 21:57:01,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 21:57:01,831 [I] Table x_portal_user does not exist in database ranger
2026-09-25 21:57:01,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 21:57:02,087 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 21:57:02,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 21:57:02,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 21:57:02,544 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 21:57:02,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 21:57:03.546 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.564 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.583 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.600 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.620 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.639 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.698 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.706 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.722 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.733 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.743 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.758 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.768 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.776 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.784 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:03.792 UTC [13274] WARNING: there is no transaction in progress
2026-09-25 21:57:04,525 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 21:57:04,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 21:57:04,757 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 21:57:04,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 21:57:05,018 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 21:57:05,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 21:57:05,266 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 21:57:05,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 21:57:05,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 21:57:05,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2g0f.c.gcp-upstream.internal';"
2026-09-25 21:57:06,020 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 21:57:06,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 21:57:06,252 [I] DB_PATCHES have already been applied
I20260925 21:57:06.258128 2331 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdR78Ha/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 21:57:06.258193 2331 mini_ranger.cc:193] Using host: 127.2.70.212
I20260925 21:57:06.261181 2331 mini_ranger.cc:227] Ranger admin URL: http://127.2.70.212:58439
Sep 25, 2026 9:57:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 9:57:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 9:57:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 9:57:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 9:57:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 9:57:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 9:57:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 21:57:07.272231 2331 mini_ranger.cc:160] Time spent starting Ranger: real 8.430s user 0.000s sys 0.008s
Sep 25, 2026 9:57:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 9:57:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 9:57:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 9:57:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 9:57:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 9:57:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 21:57:29.113286 2331 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.70.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.70.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/rangerkms_127.2.70.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/rangerkms_127.2.70.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.70.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.70.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.70.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/HTTP_127.2.70.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.70.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/HTTP_127.2.70.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 21:57:29 dist-test-slave-2g0f krb5kdc[13061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373449, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 21:57:29.195937 2331 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 21:57:29.205036 2331 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 21:57:30.113250 2331 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 21:57:30.113314 2331 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-kms
I20260925 21:57:30.113354 2331 mini_ranger_kms.cc:227] Using postgres at 127.2.70.212:47233
2026-09-25 21:57:30,285 [I] DB FLAVOR :POSTGRES
2026-09-25 21:57:30,285 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 21:57:30,285 [I] Checking connection
2026-09-25 21:57:30,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 21:57:30,580 [I] connection success
2026-09-25 21:57:30,580 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 21:57:30,580 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 21:57:30,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 21:57:30,842 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 21:57:30,842 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 21:57:30,843 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 21:57:30,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:47233/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 21:57:31,174 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 21:57:31.655735 2331 mini_ranger_kms.cc:310] Created kms service
I20260925 21:57:31.834545 2331 mini_ranger_kms.cc:326] Created kudu user
I20260925 21:57:31.877017 2331 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 21:57:31 dist-test-slave-2g0f krb5kdc[13061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373449, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:57:32.239976 2331 mini_ranger_kms.cc:384] Added ranger policy
I20260925 21:57:32.240095 2331 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 21:57:32.240118 2331 mini_ranger_kms.cc:240] Using host: 127.2.70.212
I20260925 21:57:32.243656 2331 mini_ranger_kms.cc:276] Ranger KMS PID: 13454
I20260925 21:57:32.243774 2331 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.70.212:50249
Sep 25, 2026 9:57:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 9:57:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 9:57:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 9:57:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 9:57:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 9:57:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 21:57:34.347352 2331 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.151s user 0.015s sys 0.140s
I20260925 21:57:34.347476 2331 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 21:57:34.347559 2331 mini_ranger_kms.cc:401] 127.2.70.212:50249/kms/v1/keys
Sep 25 21:57:34 dist-test-slave-2g0f krb5kdc[13061](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790373454, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.70.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 21:57:35 dist-test-slave-2g0f krb5kdc[13061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373454, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.70.212@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
Sep 25 21:57:35 dist-test-slave-2g0f krb5kdc[13061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373454, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.70.212@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
Sep 25 21:57:37 dist-test-slave-2g0f krb5kdc[13061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:57:37.795635 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:34793
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:34793
--ranger_config_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.212:50249/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:57:37.914378 13505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:57:37.914664 13505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:57:37.914695 13505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:57:37.916486 13505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:57:37.916558 13505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:57:37.916575 13505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:57:37.916592 13505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:57:37.916608 13505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:57:37.918917 13505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.70.212:50249/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:34793
--ranger_config_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:34793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:57:37.919179 13505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:57:37.919413 13505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:57:37.925637 13510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:57:37.925598 13511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:57:37.925912 13505 server_base.cc:1034] running on GCE node
W20260925 21:57:37.925614 13513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:57:37.926250 13505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:57:37.926555 13505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:57:37.927731 13505 hybrid_clock.cc:648] HybridClock initialized: now 1790373457927716 us; error 35 us; skew 500 ppm
Sep 25 21:57:37 dist-test-slave-2g0f krb5kdc[13061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373457, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:57:37.936311 13505 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:57:37.936956 13505 webserver.cc:460] Webserver started at http://127.2.70.254:40373/ using document root <none> and password file <none>
I20260925 21:57:37.937204 13505 fs_manager.cc:359] Metadata directory not provided
I20260925 21:57:37.937275 13505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:57:37.937462 13505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 21:57:37 dist-test-slave-2g0f krb5kdc[13061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373457, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:57:38.004472 13505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "4a4a34bc28aa4377aaaba0624df73d34"
format_stamp: "Formatted at 2026-09-25 21:57:37 on dist-test-slave-2g0f"
server_key: "76a6e1a1c262da6ba9ac1f69ed182180"
server_key_iv: "1734604deaef51c63c38f0400bc0a9e1"
server_key_version: "kuduclusterkey@0"
I20260925 21:57:38.004914 13505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "4a4a34bc28aa4377aaaba0624df73d34"
format_stamp: "Formatted at 2026-09-25 21:57:37 on dist-test-slave-2g0f"
server_key: "76a6e1a1c262da6ba9ac1f69ed182180"
server_key_iv: "1734604deaef51c63c38f0400bc0a9e1"
server_key_version: "kuduclusterkey@0"
I20260925 21:57:38.006253 13505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:57:38.020912 13520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:57:38.021118 13505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:57:38.021220 13505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "4a4a34bc28aa4377aaaba0624df73d34"
format_stamp: "Formatted at 2026-09-25 21:57:37 on dist-test-slave-2g0f"
server_key: "76a6e1a1c262da6ba9ac1f69ed182180"
server_key_iv: "1734604deaef51c63c38f0400bc0a9e1"
server_key_version: "kuduclusterkey@0"
I20260925 21:57:38.021319 13505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:57:38.029497 13505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:57:38.031327 13505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:57:38.031474 13505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:57:38.035881 13505 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:34793
I20260925 21:57:38.035936 13572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:34793 every 8 connection(s)
I20260925 21:57:38.036342 13505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:57:38.037005 13573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:57:38.039901 13573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34: Bootstrap starting.
I20260925 21:57:38.040539 13573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34: Neither blocks nor log segments found. Creating new log.
I20260925 21:57:38.040819 13573 log.cc:824] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34: Log is configured to *not* fsync() on all Append() calls
I20260925 21:57:38.041464 13573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34: No bootstrap required, opened a new log
I20260925 21:57:38.043030 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 13505
I20260925 21:57:38.043123 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:57:38.043324 2331 external_mini_cluster.cc:1428] Setting key 5c8ccb8be848f04183863543c7320baa
I20260925 21:57:38.043758 13573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4a34bc28aa4377aaaba0624df73d34" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34793 } }
I20260925 21:57:38.044214 13573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:57:38.044286 13573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4a34bc28aa4377aaaba0624df73d34, State: Initialized, Role: FOLLOWER
I20260925 21:57:38.044473 13573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4a34bc28aa4377aaaba0624df73d34" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34793 } }
I20260925 21:57:38.044588 13573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:57:38.044636 13573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:57:38.044699 13573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:57:38.045367 13573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4a34bc28aa4377aaaba0624df73d34" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34793 } }
I20260925 21:57:38.045469 13573 leader_election.cc:302] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a4a34bc28aa4377aaaba0624df73d34; no voters:
I20260925 21:57:38.045658 13573 leader_election.cc:288] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:57:38.045707 13578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:57:38.045918 13578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [term 1 LEADER]: Becoming Leader. State: Replica: 4a4a34bc28aa4377aaaba0624df73d34, State: Running, Role: LEADER
I20260925 21:57:38.045984 13573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:57:38.046052 13578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4a34bc28aa4377aaaba0624df73d34" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34793 } }
I20260925 21:57:38.046388 13578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a4a34bc28aa4377aaaba0624df73d34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4a34bc28aa4377aaaba0624df73d34" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34793 } } }
I20260925 21:57:38.046489 13578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [sys.catalog]: This master's current role is: LEADER
I20260925 21:57:38.046645 13578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a4a34bc28aa4377aaaba0624df73d34. Latest consensus state: current_term: 1 leader_uuid: "4a4a34bc28aa4377aaaba0624df73d34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4a34bc28aa4377aaaba0624df73d34" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34793 } } }
I20260925 21:57:38.046715 13578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34 [sys.catalog]: This master's current role is: LEADER
I20260925 21:57:38.047576 13573 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 25 21:57:38 dist-test-slave-2g0f krb5kdc[13061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:57:38.052793 13584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:57:38.053450 13584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:57:38.057546 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:38.043894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.2.70.254:34793)
0925 21:57:38.044077 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 21:57:38.044088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 21:57:38.044122 (+ 34us) server_negotiation.cc:407] Connection header received
0925 21:57:38.044241 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:57:38.044249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:57:38.044310 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:57:38.044456 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:57:38.045171 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:38.046554 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:38.047664 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:38.050419 (+ 2755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:38.050613 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:38.053538 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:57:38.053542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:57:38.053543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:57:38.053556 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:57:38.055554 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:57:38.055989 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:57:38.055992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:57:38.055992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:57:38.056107 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:57:38.056299 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:57:38.056299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:57:38.056300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:57:38.056620 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:57:38.057094 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0925 21:57:38.057166 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 21:57:38.057300 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260925 21:57:38.059527 13584 catalog_manager.cc:1347] Generated new cluster ID: bf9a796002474bf6a5e778746b7e0efe
I20260925 21:57:38.059580 13584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:57:38.069585 13584 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:57:38.070431 13584 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:57:38.080483 13584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34: Generated new TSK 0
I20260925 21:57:38.080794 13584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:57:38.874822 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.212:50249/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34793
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:57:38.994891 13618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:57:38.995246 13618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:57:38.995327 13618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:57:38.997112 13618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:57:38.997231 13618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:57:38.997334 13618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:57:38.999600 13618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.70.212:50249/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:57:38.999915 13618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:57:39.000240 13618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:57:39.006538 13626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:57:39.006709 13624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:57:39.006934 13623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:57:39.008030 13618 server_base.cc:1034] running on GCE node
I20260925 21:57:39.008255 13618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:57:39.008543 13618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:57:39.009708 13618 hybrid_clock.cc:648] HybridClock initialized: now 1790373459009685 us; error 38 us; skew 500 ppm
Sep 25 21:57:39 dist-test-slave-2g0f krb5kdc[13061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:57:39.013155 13618 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:57:39.013701 13618 webserver.cc:460] Webserver started at http://127.2.70.193:33075/ using document root <none> and password file <none>
I20260925 21:57:39.013967 13618 fs_manager.cc:359] Metadata directory not provided
I20260925 21:57:39.014045 13618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:57:39.014179 13618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 21:57:39 dist-test-slave-2g0f krb5kdc[13061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:57:39.028795 13618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "0facef5c76e8412fbfc8d98876a1ad18"
format_stamp: "Formatted at 2026-09-25 21:57:39 on dist-test-slave-2g0f"
server_key: "4dd6bfacb8a8c6c61750f6a7325a2145"
server_key_iv: "d470f364ac208cdcf9fc43cf9dc6a3d6"
server_key_version: "kuduclusterkey@0"
I20260925 21:57:39.029394 13618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "0facef5c76e8412fbfc8d98876a1ad18"
format_stamp: "Formatted at 2026-09-25 21:57:39 on dist-test-slave-2g0f"
server_key: "4dd6bfacb8a8c6c61750f6a7325a2145"
server_key_iv: "d470f364ac208cdcf9fc43cf9dc6a3d6"
server_key_version: "kuduclusterkey@0"
I20260925 21:57:39.031401 13618 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 21:57:39.054396 13635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:57:39.054610 13618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:57:39.054709 13618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "0facef5c76e8412fbfc8d98876a1ad18"
format_stamp: "Formatted at 2026-09-25 21:57:39 on dist-test-slave-2g0f"
server_key: "4dd6bfacb8a8c6c61750f6a7325a2145"
server_key_iv: "d470f364ac208cdcf9fc43cf9dc6a3d6"
server_key_version: "kuduclusterkey@0"
I20260925 21:57:39.054806 13618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:57:39.065912 13618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:57:39.068214 13618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:57:39.068408 13618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:57:39.068701 13618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:57:39.069103 13618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:57:39.069193 13618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:57:39.069289 13618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:57:39.069362 13618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:57:39.075750 13618 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:40421
I20260925 21:57:39.076294 13618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:57:39.078156 13748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:40421 every 8 connection(s)
I20260925 21:57:39.082932 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 13618
I20260925 21:57:39.083037 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:57:39.083225 2331 external_mini_cluster.cc:1428] Setting key 67fc95869282ecec3d7adc8d18700b6f
Sep 25 21:57:39 dist-test-slave-2g0f krb5kdc[13061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:57:39.092103 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.077277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:51129 (local address 127.2.70.254:34793)
0925 21:57:39.079408 (+ 2131us) server_negotiation.cc:206] Beginning negotiation
0925 21:57:39.079410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:57:39.079419 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:57:39.079446 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:57:39.079447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:57:39.079458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:57:39.079534 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:57:39.080321 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.081266 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.085398 (+ 4132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.085734 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.085802 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.088421 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:57:39.088423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:57:39.088423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:57:39.088431 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:57:39.090210 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:57:39.090733 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:57:39.090734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:57:39.090735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:57:39.090803 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:57:39.091055 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:57:39.091055 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:57:39.091056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:57:39.091195 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:57:39.091247 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 21:57:39.091905 (+ 658us) server_negotiation.cc:299] Negotiation successful
0925 21:57:39.091993 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2100,"thread_start_us":71,"threads_started":1}
I20260925 21:57:39.092636 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.077033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34793 (local address 127.2.70.193:51129)
0925 21:57:39.078557 (+ 1524us) negotiation.cc:107] Waiting for socket to connect
0925 21:57:39.078602 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 21:57:39.079221 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:57:39.079669 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:57:39.079673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:57:39.079931 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:57:39.080209 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.080215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.081369 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.081370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.082656 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.082659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.085875 (+ 3216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.085876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.086032 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.086605 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:57:39.086615 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:57:39.088309 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:57:39.090302 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:57:39.090310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:57:39.090313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:57:39.090647 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:57:39.090864 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:57:39.090864 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:57:39.090865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:57:39.090994 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:57:39.091270 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:57:39.091275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:57:39.091443 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 21:57:39.091559 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 21:57:39.091718 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1479,"thread_start_us":84,"threads_started":1}
I20260925 21:57:39.093367 13749 heartbeater.cc:347] Connected to a master server at 127.2.70.254:34793
I20260925 21:57:39.093519 13749 heartbeater.cc:464] Registering TS with master...
I20260925 21:57:39.093775 13749 heartbeater.cc:511] Master 127.2.70.254:34793 requested a full tablet report, sending...
I20260925 21:57:39.094398 13537 ts_manager.cc:194] Registered new tserver with Master: 0facef5c76e8412fbfc8d98876a1ad18 (127.2.70.193:40421)
I20260925 21:57:39.095009 13537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:51129
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:57:39.128293 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.212:50249/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34793
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 21:57:39 dist-test-slave-2g0f krb5kdc[13061](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790373459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 21:57:39.259882 13755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:57:39.260239 13755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:57:39.260344 13755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:57:39.262298 13755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:57:39.262424 13755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:57:39.262521 13755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:57:39.265034 13755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.70.212:50249/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:57:39.265738 13755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:57:39.266106 13755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:57:39.273608 13762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:57:39.274029 13755 server_base.cc:1034] running on GCE node
W20260925 21:57:39.274497 13765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:57:39.276345 13763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:57:39.276693 13755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:57:39.276962 13755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:57:39.280203 13755 hybrid_clock.cc:648] HybridClock initialized: now 1790373459280189 us; error 39 us; skew 500 ppm
Sep 25 21:57:39 dist-test-slave-2g0f krb5kdc[13061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:57:39.283633 13755 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:57:39.284255 13755 webserver.cc:460] Webserver started at http://127.2.70.194:33343/ using document root <none> and password file <none>
I20260925 21:57:39.284533 13755 fs_manager.cc:359] Metadata directory not provided
I20260925 21:57:39.284590 13755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:57:39.284724 13755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 21:57:39 dist-test-slave-2g0f krb5kdc[13061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:57:39.303122 13755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "b95d92fb88074f97b71eb6fce1602594"
format_stamp: "Formatted at 2026-09-25 21:57:39 on dist-test-slave-2g0f"
server_key: "5d6dd97376f57c902ee2a42ab19b642c"
server_key_iv: "e06768806c1559d17e25de60a73710a6"
server_key_version: "kuduclusterkey@0"
I20260925 21:57:39.303730 13755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "b95d92fb88074f97b71eb6fce1602594"
format_stamp: "Formatted at 2026-09-25 21:57:39 on dist-test-slave-2g0f"
server_key: "5d6dd97376f57c902ee2a42ab19b642c"
server_key_iv: "e06768806c1559d17e25de60a73710a6"
server_key_version: "kuduclusterkey@0"
I20260925 21:57:39.308907 13755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260925 21:57:39.327777 13772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:57:39.328099 13755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 21:57:39.328225 13755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "b95d92fb88074f97b71eb6fce1602594"
format_stamp: "Formatted at 2026-09-25 21:57:39 on dist-test-slave-2g0f"
server_key: "5d6dd97376f57c902ee2a42ab19b642c"
server_key_iv: "e06768806c1559d17e25de60a73710a6"
server_key_version: "kuduclusterkey@0"
I20260925 21:57:39.328343 13755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:57:39.357470 13755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:57:39.361325 13755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:57:39.361549 13755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:57:39.361852 13755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:57:39.362272 13755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:57:39.362361 13755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:57:39.362428 13755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:57:39.362483 13755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:57:39.369151 13755 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:42127
I20260925 21:57:39.369632 13755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:57:39.375283 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 13755
I20260925 21:57:39.375398 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:57:39.375622 2331 external_mini_cluster.cc:1428] Setting key 7747f3595cdf56ba04c88e009bb14e06
I20260925 21:57:39.378172 13886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:42127 every 8 connection(s)
Sep 25 21:57:39 dist-test-slave-2g0f krb5kdc[13061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
I20260925 21:57:39.389292 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.378892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:44959 (local address 127.2.70.254:34793)
0925 21:57:39.379167 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 21:57:39.379169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:57:39.379204 (+ 35us) server_negotiation.cc:407] Connection header received
0925 21:57:39.379872 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:57:39.379874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:57:39.379890 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:57:39.379918 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:57:39.380741 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.381525 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.383102 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.383413 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.383468 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.385951 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:57:39.385952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:57:39.385953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:57:39.385960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:57:39.387416 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:57:39.388069 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:57:39.388070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:57:39.388070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:57:39.388132 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:57:39.388416 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:57:39.388417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:57:39.388418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:57:39.388633 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:57:39.388681 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 21:57:39.389059 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 21:57:39.389189 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
I20260925 21:57:39.391952 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.378889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34793 (local address 127.2.70.194:44959)
0925 21:57:39.379081 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 21:57:39.379123 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 21:57:39.379719 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:57:39.380114 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:57:39.380119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:57:39.380345 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:57:39.380628 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.380634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.381645 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.381646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.383004 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.383007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.383564 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.383565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.383726 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.384261 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:57:39.384271 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:57:39.385860 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:57:39.387642 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:57:39.387649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:57:39.387651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:57:39.387969 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:57:39.388205 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:57:39.388205 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:57:39.388206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:57:39.388339 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:57:39.388728 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:57:39.388734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:57:39.388899 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 21:57:39.391467 (+ 2568us) client_negotiation.cc:240] Negotiation successful
0925 21:57:39.391663 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260925 21:57:39.392663 13887 heartbeater.cc:347] Connected to a master server at 127.2.70.254:34793
I20260925 21:57:39.392762 13887 heartbeater.cc:464] Registering TS with master...
I20260925 21:57:39.392956 13887 heartbeater.cc:511] Master 127.2.70.254:34793 requested a full tablet report, sending...
I20260925 21:57:39.393393 13537 ts_manager.cc:194] Registered new tserver with Master: b95d92fb88074f97b71eb6fce1602594 (127.2.70.194:42127)
I20260925 21:57:39.393908 13537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:44959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:57:39.411002 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.212:50249/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34793
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:57:39.529687 13892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:57:39.530051 13892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:57:39.530136 13892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:57:39.531946 13892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:57:39.532068 13892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:57:39.532150 13892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:57:39.534493 13892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.70.212:50249/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:57:39.534986 13892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:57:39.535281 13892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:57:39.541258 13898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:57:39.541613 13900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:57:39.541846 13897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:57:39.541857 13892 server_base.cc:1034] running on GCE node
I20260925 21:57:39.542152 13892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:57:39.542402 13892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:57:39.545647 13892 hybrid_clock.cc:648] HybridClock initialized: now 1790373459545621 us; error 36 us; skew 500 ppm
Sep 25 21:57:39 dist-test-slave-2g0f krb5kdc[13061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:57:39.549551 13892 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:57:39.550122 13892 webserver.cc:460] Webserver started at http://127.2.70.195:34665/ using document root <none> and password file <none>
I20260925 21:57:39.550413 13892 fs_manager.cc:359] Metadata directory not provided
I20260925 21:57:39.550482 13892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:57:39.550700 13892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 21:57:39 dist-test-slave-2g0f krb5kdc[13061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:57:39.577219 13892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "5903194f837441c29b1ff59121ae5741"
format_stamp: "Formatted at 2026-09-25 21:57:39 on dist-test-slave-2g0f"
server_key: "0c582722cc06af98fde47c5fed47453e"
server_key_iv: "3b549db17eaff073c51186586f90b837"
server_key_version: "kuduclusterkey@0"
I20260925 21:57:39.577711 13892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "5903194f837441c29b1ff59121ae5741"
format_stamp: "Formatted at 2026-09-25 21:57:39 on dist-test-slave-2g0f"
server_key: "0c582722cc06af98fde47c5fed47453e"
server_key_iv: "3b549db17eaff073c51186586f90b837"
server_key_version: "kuduclusterkey@0"
I20260925 21:57:39.579591 13892 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 21:57:39.592443 13907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:57:39.593225 13892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:57:39.593394 13892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "5903194f837441c29b1ff59121ae5741"
format_stamp: "Formatted at 2026-09-25 21:57:39 on dist-test-slave-2g0f"
server_key: "0c582722cc06af98fde47c5fed47453e"
server_key_iv: "3b549db17eaff073c51186586f90b837"
server_key_version: "kuduclusterkey@0"
I20260925 21:57:39.593573 13892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:57:39.665542 13892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:57:39.667961 13892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:57:39.668169 13892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:57:39.668490 13892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:57:39.668913 13892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:57:39.669004 13892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:57:39.669073 13892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:57:39.669124 13892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:57:39.675416 13892 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:40201
I20260925 21:57:39.675910 13892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:57:39.683668 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 13892
I20260925 21:57:39.683781 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:57:39.684090 2331 external_mini_cluster.cc:1428] Setting key 26720d08e62c85b2d7ce5675c76d6f14
I20260925 21:57:39.678489 14020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:40201 every 8 connection(s)
Sep 25 21:57:39 dist-test-slave-2g0f krb5kdc[13061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:57:39.697715 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.679161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:54663 (local address 127.2.70.254:34793)
0925 21:57:39.679207 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 21:57:39.679209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:57:39.686880 (+ 7671us) server_negotiation.cc:407] Connection header received
0925 21:57:39.687559 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:57:39.687560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:57:39.687573 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:57:39.687602 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:57:39.688564 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.689350 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.690966 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.691308 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.691349 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.693987 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:57:39.693989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:57:39.693989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:57:39.693997 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:57:39.695576 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:57:39.696687 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:57:39.696689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:57:39.696689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:57:39.696749 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:57:39.696994 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:57:39.696995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:57:39.696996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:57:39.697128 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:57:39.697173 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 21:57:39.697535 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 21:57:39.697619 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 21:57:39.698608 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.679146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34793 (local address 127.2.70.195:54663)
0925 21:57:39.686724 (+ 7578us) negotiation.cc:107] Waiting for socket to connect
0925 21:57:39.686793 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 21:57:39.687438 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:57:39.687795 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:57:39.687800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:57:39.688105 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:57:39.688389 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.688395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.689582 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.689582 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.690867 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.690870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.691439 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.691440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.691596 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.692228 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:57:39.692239 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:57:39.693877 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:57:39.696275 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:57:39.696283 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:57:39.696286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:57:39.696602 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:57:39.696808 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:57:39.696808 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:57:39.696809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:57:39.696931 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:57:39.697202 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:57:39.697208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:57:39.697383 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 21:57:39.698179 (+ 796us) client_negotiation.cc:240] Negotiation successful
0925 21:57:39.698360 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7520,"thread_start_us":83,"threads_started":1}
I20260925 21:57:39.699198 14021 heartbeater.cc:347] Connected to a master server at 127.2.70.254:34793
I20260925 21:57:39.699303 14021 heartbeater.cc:464] Registering TS with master...
I20260925 21:57:39.699498 14021 heartbeater.cc:511] Master 127.2.70.254:34793 requested a full tablet report, sending...
I20260925 21:57:39.699908 13537 ts_manager.cc:194] Registered new tserver with Master: 5903194f837441c29b1ff59121ae5741 (127.2.70.195:40201)
I20260925 21:57:39.700393 13537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:54663
I20260925 21:57:39.706899 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:57:39.716643 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.708622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.2.70.254:34793)
0925 21:57:39.708666 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 21:57:39.708668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:57:39.708949 (+ 281us) server_negotiation.cc:407] Connection header received
0925 21:57:39.709043 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:57:39.709044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:57:39.709054 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:57:39.709082 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:57:39.709800 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.710601 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.711714 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.712032 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.712073 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.713702 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:57:39.713704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:57:39.713704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:57:39.713712 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:57:39.715374 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:57:39.715757 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:57:39.715758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:57:39.715758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:57:39.715812 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:57:39.716056 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:57:39.716057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:57:39.716058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:57:39.716184 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:57:39.716255 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 21:57:39.716466 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 21:57:39.716545 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 21:57:39.718827 13537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:57:39.720996 13537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:57:39.734130 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.724876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59034 (local address 127.2.70.193:40421)
0925 21:57:39.725385 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 21:57:39.725390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:57:39.725798 (+ 408us) server_negotiation.cc:407] Connection header received
0925 21:57:39.725907 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:57:39.725910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:57:39.725926 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:57:39.725976 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:57:39.726216 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.727870 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.729529 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.731797 (+ 2268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.731893 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.733369 (+ 1476us) server_negotiation.cc:1035] Waiting for connection context
0925 21:57:39.733988 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 21:57:39.734041 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":65,"threads_started":1}
I20260925 21:57:39.735636 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.724628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:40421 (local address 127.0.0.1:59034)
0925 21:57:39.725718 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0925 21:57:39.725726 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:57:39.725833 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:57:39.726077 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:57:39.726079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:57:39.726081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:57:39.726147 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.726148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.727961 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.727962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.729194 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.729459 (+ 265us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.733422 (+ 3963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.733423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.733516 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.733530 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 21:57:39.734608 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0925 21:57:39.735559 (+ 951us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"thread_start_us":59,"threads_started":1}
I20260925 21:57:39.736596 13669 tablet_service.cc:1467] Processing CreateTablet for tablet e95dadc902af4bf7a43879dd1740af14 (DEFAULT_TABLE table=test-table [id=08fc9890c73e4869a9475430d52a8cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:57:39.736964 13669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95dadc902af4bf7a43879dd1740af14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:57:39.740907 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.724431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:42127 (local address 127.0.0.1:37346)
0925 21:57:39.725449 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0925 21:57:39.725491 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 21:57:39.725554 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:57:39.734523 (+ 8969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:57:39.734524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:57:39.734527 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:57:39.734570 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.734571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.737228 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.737229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.738181 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.738184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.740657 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.740658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.740733 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.740741 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 21:57:39.740772 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 21:57:39.740826 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":67,"threads_started":1}
I20260925 21:57:39.740933 14039 tablet_bootstrap.cc:492] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18: Bootstrap starting.
I20260925 21:57:39.742175 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.724760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:40201 (local address 127.0.0.1:35834)
0925 21:57:39.731010 (+ 6250us) negotiation.cc:107] Waiting for socket to connect
0925 21:57:39.731022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:57:39.731083 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:57:39.731761 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:57:39.731762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:57:39.731765 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:57:39.731824 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.731847 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.735090 (+ 3243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.735091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.740741 (+ 5650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.740745 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.741924 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.741925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.742006 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.742011 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:57:39.742058 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 21:57:39.742099 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6229,"thread_start_us":68,"threads_started":1}
I20260925 21:57:39.742198 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.725055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.2.70.195:40201)
0925 21:57:39.725679 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0925 21:57:39.725683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:57:39.731284 (+ 5601us) server_negotiation.cc:407] Connection header received
0925 21:57:39.731337 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:57:39.731339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:57:39.731357 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:57:39.731437 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:57:39.732210 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.733266 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.741037 (+ 7771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.741820 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.741889 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.741961 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 21:57:39.742082 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 21:57:39.742120 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":77,"threads_started":1}
I20260925 21:57:39.742517 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.724526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.2.70.194:42127)
0925 21:57:39.733670 (+ 9144us) server_negotiation.cc:206] Beginning negotiation
0925 21:57:39.733675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:57:39.733684 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:57:39.733720 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:57:39.733722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:57:39.733737 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:57:39.735152 (+ 1415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:57:39.735236 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.737113 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.738298 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.738989 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.742008 (+ 3019us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.742372 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0925 21:57:39.742399 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 21:57:39.742443 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9106,"thread_start_us":83,"threads_started":1}
I20260925 21:57:39.743258 13955 tablet_service.cc:1467] Processing CreateTablet for tablet e95dadc902af4bf7a43879dd1740af14 (DEFAULT_TABLE table=test-table [id=08fc9890c73e4869a9475430d52a8cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:57:39.743592 13955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95dadc902af4bf7a43879dd1740af14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:57:39.744379 13821 tablet_service.cc:1467] Processing CreateTablet for tablet e95dadc902af4bf7a43879dd1740af14 (DEFAULT_TABLE table=test-table [id=08fc9890c73e4869a9475430d52a8cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:57:39.744683 13821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95dadc902af4bf7a43879dd1740af14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:57:39.751672 14039 tablet_bootstrap.cc:654] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18: Neither blocks nor log segments found. Creating new log.
I20260925 21:57:39.752136 14039 log.cc:824] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18: Log is configured to *not* fsync() on all Append() calls
I20260925 21:57:39.752543 14040 tablet_bootstrap.cc:492] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594: Bootstrap starting.
I20260925 21:57:39.752799 14041 tablet_bootstrap.cc:492] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741: Bootstrap starting.
I20260925 21:57:39.753804 14041 tablet_bootstrap.cc:654] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741: Neither blocks nor log segments found. Creating new log.
I20260925 21:57:39.755246 14040 tablet_bootstrap.cc:654] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594: Neither blocks nor log segments found. Creating new log.
I20260925 21:57:39.755606 14041 log.cc:824] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741: Log is configured to *not* fsync() on all Append() calls
I20260925 21:57:39.756201 14040 log.cc:824] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594: Log is configured to *not* fsync() on all Append() calls
I20260925 21:57:39.757157 14039 tablet_bootstrap.cc:492] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18: No bootstrap required, opened a new log
I20260925 21:57:39.757274 14039 ts_tablet_manager.cc:1397] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260925 21:57:39.759912 14039 raft_consensus.cc:348] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95d92fb88074f97b71eb6fce1602594" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42127 } } peers { permanent_uuid: "5903194f837441c29b1ff59121ae5741" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40201 } } peers { permanent_uuid: "0facef5c76e8412fbfc8d98876a1ad18" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40421 } }
I20260925 21:57:39.760213 14039 raft_consensus.cc:374] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:57:39.760290 14039 raft_consensus.cc:729] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0facef5c76e8412fbfc8d98876a1ad18, State: Initialized, Role: FOLLOWER
I20260925 21:57:39.760494 14039 consensus_queue.cc:260] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95d92fb88074f97b71eb6fce1602594" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42127 } } peers { permanent_uuid: "5903194f837441c29b1ff59121ae5741" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40201 } } peers { permanent_uuid: "0facef5c76e8412fbfc8d98876a1ad18" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40421 } }
I20260925 21:57:39.760848 14039 ts_tablet_manager.cc:1428] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:57:39.761049 13749 heartbeater.cc:503] Master 127.2.70.254:34793 was elected leader, sending a full tablet report...
I20260925 21:57:39.762967 14041 tablet_bootstrap.cc:492] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741: No bootstrap required, opened a new log
I20260925 21:57:39.763095 14041 ts_tablet_manager.cc:1397] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260925 21:57:39.765085 14041 raft_consensus.cc:348] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95d92fb88074f97b71eb6fce1602594" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42127 } } peers { permanent_uuid: "5903194f837441c29b1ff59121ae5741" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40201 } } peers { permanent_uuid: "0facef5c76e8412fbfc8d98876a1ad18" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40421 } }
I20260925 21:57:39.765321 14041 raft_consensus.cc:374] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:57:39.765393 14041 raft_consensus.cc:729] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5903194f837441c29b1ff59121ae5741, State: Initialized, Role: FOLLOWER
I20260925 21:57:39.765580 14041 consensus_queue.cc:260] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95d92fb88074f97b71eb6fce1602594" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42127 } } peers { permanent_uuid: "5903194f837441c29b1ff59121ae5741" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40201 } } peers { permanent_uuid: "0facef5c76e8412fbfc8d98876a1ad18" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40421 } }
I20260925 21:57:39.765903 14041 ts_tablet_manager.cc:1428] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 21:57:39.766047 14021 heartbeater.cc:503] Master 127.2.70.254:34793 was elected leader, sending a full tablet report...
I20260925 21:57:39.767321 14040 tablet_bootstrap.cc:492] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594: No bootstrap required, opened a new log
I20260925 21:57:39.767453 14040 ts_tablet_manager.cc:1397] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260925 21:57:39.769769 14040 raft_consensus.cc:348] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95d92fb88074f97b71eb6fce1602594" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42127 } } peers { permanent_uuid: "5903194f837441c29b1ff59121ae5741" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40201 } } peers { permanent_uuid: "0facef5c76e8412fbfc8d98876a1ad18" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40421 } }
I20260925 21:57:39.770017 14040 raft_consensus.cc:374] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:57:39.770092 14040 raft_consensus.cc:729] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95d92fb88074f97b71eb6fce1602594, State: Initialized, Role: FOLLOWER
I20260925 21:57:39.770273 14040 consensus_queue.cc:260] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95d92fb88074f97b71eb6fce1602594" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42127 } } peers { permanent_uuid: "5903194f837441c29b1ff59121ae5741" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40201 } } peers { permanent_uuid: "0facef5c76e8412fbfc8d98876a1ad18" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40421 } }
I20260925 21:57:39.770618 14040 ts_tablet_manager.cc:1428] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:57:39.770771 13887 heartbeater.cc:503] Master 127.2.70.254:34793 was elected leader, sending a full tablet report...
W20260925 21:57:39.829689 13750 tablet.cc:2369] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 21:57:39 dist-test-slave-2g0f krb5kdc[13061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
W20260925 21:57:39.879637 13888 tablet.cc:2369] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 21:57:39.929126 14022 tablet.cc:2369] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:57:39.936604 14047 raft_consensus.cc:482] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:57:39.936748 14047 raft_consensus.cc:504] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95d92fb88074f97b71eb6fce1602594" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42127 } } peers { permanent_uuid: "5903194f837441c29b1ff59121ae5741" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40201 } } peers { permanent_uuid: "0facef5c76e8412fbfc8d98876a1ad18" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40421 } }
I20260925 21:57:39.937142 14047 leader_election.cc:288] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5903194f837441c29b1ff59121ae5741 (127.2.70.195:40201), 0facef5c76e8412fbfc8d98876a1ad18 (127.2.70.193:40421)
I20260925 21:57:39.941777 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.938026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:41561 (local address 127.2.70.195:40201)
0925 21:57:39.938115 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 21:57:39.938117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:57:39.938272 (+ 155us) server_negotiation.cc:407] Connection header received
0925 21:57:39.938374 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:57:39.938375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:57:39.938388 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:57:39.938415 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:57:39.938626 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.939309 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.940750 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.941238 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.941287 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.941296 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:57:39.941664 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 21:57:39.941707 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 21:57:39.942225 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.937444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:40201 (local address 127.2.70.194:41561)
0925 21:57:39.938204 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 21:57:39.938215 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:57:39.938302 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:57:39.938489 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:57:39.938490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:57:39.938492 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:57:39.938561 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.938564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.939430 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.939431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.940639 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.940643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.941362 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.941362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.941574 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.941581 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 21:57:39.942099 (+ 518us) client_negotiation.cc:240] Negotiation successful
0925 21:57:39.942144 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":62,"threads_started":1}
I20260925 21:57:39.942698 13972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95dadc902af4bf7a43879dd1740af14" candidate_uuid: "b95d92fb88074f97b71eb6fce1602594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5903194f837441c29b1ff59121ae5741" is_pre_election: true
I20260925 21:57:39.942888 13972 raft_consensus.cc:2454] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95d92fb88074f97b71eb6fce1602594 in term 0.
I20260925 21:57:39.943222 13774 leader_election.cc:302] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5903194f837441c29b1ff59121ae5741, b95d92fb88074f97b71eb6fce1602594; no voters:
I20260925 21:57:39.943357 14047 raft_consensus.cc:2781] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:57:39.943400 14047 raft_consensus.cc:482] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:57:39.943421 14047 raft_consensus.cc:3037] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:57:39.943434 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.938070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:54773 (local address 127.2.70.193:40421)
0925 21:57:39.938149 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 21:57:39.938151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:57:39.940009 (+ 1858us) server_negotiation.cc:407] Connection header received
0925 21:57:39.940100 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:57:39.940101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:57:39.940113 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:57:39.940140 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:57:39.940313 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.941215 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.942509 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.943079 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.943122 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.943129 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 21:57:39.943325 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 21:57:39.943365 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 21:57:39.943624 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:39.937632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:40421 (local address 127.2.70.194:54773)
0925 21:57:39.939933 (+ 2301us) negotiation.cc:107] Waiting for socket to connect
0925 21:57:39.939945 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:57:39.940035 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:57:39.940196 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:57:39.940197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:57:39.940198 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:57:39.940251 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.940252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.941310 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.941311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.942409 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:57:39.942411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:57:39.943181 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:57:39.943182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:57:39.943245 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:39.943249 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:57:39.943509 (+ 260us) client_negotiation.cc:240] Negotiation successful
0925 21:57:39.943546 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2283,"thread_start_us":40,"threads_started":1}
I20260925 21:57:39.944387 14047 raft_consensus.cc:504] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95d92fb88074f97b71eb6fce1602594" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42127 } } peers { permanent_uuid: "5903194f837441c29b1ff59121ae5741" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40201 } } peers { permanent_uuid: "0facef5c76e8412fbfc8d98876a1ad18" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40421 } }
I20260925 21:57:39.944519 14047 leader_election.cc:288] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [CANDIDATE]: Term 1 election: Requested vote from peers 5903194f837441c29b1ff59121ae5741 (127.2.70.195:40201), 0facef5c76e8412fbfc8d98876a1ad18 (127.2.70.193:40421)
I20260925 21:57:39.944943 13687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95dadc902af4bf7a43879dd1740af14" candidate_uuid: "b95d92fb88074f97b71eb6fce1602594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0facef5c76e8412fbfc8d98876a1ad18" is_pre_election: true
I20260925 21:57:39.945122 13687 raft_consensus.cc:2454] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95d92fb88074f97b71eb6fce1602594 in term 0.
I20260925 21:57:39.945250 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95dadc902af4bf7a43879dd1740af14" candidate_uuid: "b95d92fb88074f97b71eb6fce1602594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0facef5c76e8412fbfc8d98876a1ad18"
I20260925 21:57:39.945312 13703 raft_consensus.cc:3037] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:57:39.946161 13703 raft_consensus.cc:2454] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95d92fb88074f97b71eb6fce1602594 in term 1.
I20260925 21:57:39.946344 13775 leader_election.cc:302] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0facef5c76e8412fbfc8d98876a1ad18, b95d92fb88074f97b71eb6fce1602594; no voters:
I20260925 21:57:39.946499 13972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95dadc902af4bf7a43879dd1740af14" candidate_uuid: "b95d92fb88074f97b71eb6fce1602594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5903194f837441c29b1ff59121ae5741"
I20260925 21:57:39.946587 13972 raft_consensus.cc:3037] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:57:39.947425 13972 raft_consensus.cc:2454] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95d92fb88074f97b71eb6fce1602594 in term 1.
I20260925 21:57:39.947602 14047 raft_consensus.cc:2781] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:57:39.947764 14047 raft_consensus.cc:686] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [term 1 LEADER]: Becoming Leader. State: Replica: b95d92fb88074f97b71eb6fce1602594, State: Running, Role: LEADER
I20260925 21:57:39.947893 14047 consensus_queue.cc:237] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95d92fb88074f97b71eb6fce1602594" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42127 } } peers { permanent_uuid: "5903194f837441c29b1ff59121ae5741" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40201 } } peers { permanent_uuid: "0facef5c76e8412fbfc8d98876a1ad18" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40421 } }
I20260925 21:57:39.948757 13536 catalog_manager.cc:5495] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 reported cstate change: term changed from 0 to 1, leader changed from <none> to b95d92fb88074f97b71eb6fce1602594 (127.2.70.194). New cstate: current_term: 1 leader_uuid: "b95d92fb88074f97b71eb6fce1602594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95d92fb88074f97b71eb6fce1602594" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5903194f837441c29b1ff59121ae5741" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0facef5c76e8412fbfc8d98876a1ad18" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40421 } health_report { overall_health: UNKNOWN } } }
I20260925 21:57:40.010674 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:40.007357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.2.70.194:42127)
0925 21:57:40.007418 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 21:57:40.007420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:57:40.007431 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:57:40.007504 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:57:40.007505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:57:40.007529 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:57:40.007600 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:57:40.007770 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:40.008563 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:40.009672 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:57:40.009991 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:57:40.010069 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:57:40.010249 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:57:40.010395 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:57:40.010491 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 21:57:40.010550 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 21:57:40.010576 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 21:57:40.012717 13703 raft_consensus.cc:1261] T e95dadc902af4bf7a43879dd1740af14 P 0facef5c76e8412fbfc8d98876a1ad18 [term 1 FOLLOWER]: Refusing update from remote peer b95d92fb88074f97b71eb6fce1602594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:57:40.013170 13972 raft_consensus.cc:1261] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741 [term 1 FOLLOWER]: Refusing update from remote peer b95d92fb88074f97b71eb6fce1602594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:57:40.013144 14047 consensus_queue.cc:1035] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0facef5c76e8412fbfc8d98876a1ad18" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:57:40.014618 14047 consensus_queue.cc:1035] T e95dadc902af4bf7a43879dd1740af14 P b95d92fb88074f97b71eb6fce1602594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5903194f837441c29b1ff59121ae5741" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 40201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:57:40.016042 14054 mvcc.cc:205] Tried to move back new op lower bound from 7333369692209500160 to 7333369691947257856. Current Snapshot: MvccSnapshot[applied={T|T < 7333369692209500160}]
I20260925 21:57:40.017582 14053 mvcc.cc:205] Tried to move back new op lower bound from 7333369692209500160 to 7333369691947257856. Current Snapshot: MvccSnapshot[applied={T|T < 7333369692209500160}]
I20260925 21:57:40.020510 13536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:57:40.020601 13536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:57:40.021345 13536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a4a34bc28aa4377aaaba0624df73d34: Sending DeleteTablet for 3 replicas of tablet e95dadc902af4bf7a43879dd1740af14
I20260925 21:57:40.021765 13821 tablet_service.cc:1514] Processing DeleteTablet for tablet e95dadc902af4bf7a43879dd1740af14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:57:40 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:37346
I20260925 21:57:40.021924 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 13618
I20260925 21:57:40.021992 13669 tablet_service.cc:1514] Processing DeleteTablet for tablet e95dadc902af4bf7a43879dd1740af14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:57:40 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:59034
I20260925 21:57:40.023768 13955 tablet_service.cc:1514] Processing DeleteTablet for tablet e95dadc902af4bf7a43879dd1740af14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:57:40 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:35834
I20260925 21:57:40.024322 14058 mvcc.cc:205] Tried to move back new op lower bound from 7333369692209500160 to 7333369691947257856. Current Snapshot: MvccSnapshot[applied={T|T < 7333369692209500160}]
W20260925 21:57:40.028192 13523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:40421 (error 108)
I20260925 21:57:40.028868 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 13755
I20260925 21:57:40.032013 14062 tablet_replica.cc:330] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741: stopping tablet replica
I20260925 21:57:40.032143 14062 raft_consensus.cc:2229] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:57:40.032212 14062 raft_consensus.cc:2258] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:57:40.033207 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:40.033072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:40421 (local address 127.0.0.1:59042)
0925 21:57:40.033122 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 21:57:40.033171 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:40421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 21:57:40.033354 13523 catalog_manager.cc:4584] TS 0facef5c76e8412fbfc8d98876a1ad18 (127.2.70.193:40421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e95dadc902af4bf7a43879dd1740af14: Network error: Client connection negotiation failed: client connection to 127.2.70.193:40421: connect: Connection refused (error 111)
I20260925 21:57:40.033635 14062 ts_tablet_manager.cc:1905] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:57:40.035580 14062 ts_tablet_manager.cc:1918] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:57:40.035673 14062 log.cc:1196] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/e95dadc902af4bf7a43879dd1740af14
W20260925 21:57:40.036082 14033 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:57:40.035575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:42127 (local address 127.0.0.1:37364)
0925 21:57:40.035617 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 21:57:40.035627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:57:40.035663 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:57:40.036014 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.194:42127: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 21:57:40.036276 14062 ts_tablet_manager.cc:1939] T e95dadc902af4bf7a43879dd1740af14 P 5903194f837441c29b1ff59121ae5741: Deleting consensus metadata
I20260925 21:57:40.036633 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 13892
I20260925 21:57:40.040720 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:57:40.040611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:40201 (local address 127.0.0.1:35842)
0925 21:57:40.040662 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 21:57:40.040685 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.195:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 21:57:40.040755 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 13505
I20260925 21:57:40.046645 2331 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 21:57:40.381165 2331 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 21:57:40.381274 2331 mini_ranger.cc:66] Stopping Ranger...
I20260925 21:57:40.715878 2331 mini_ranger.cc:68] Stopped Ranger
2026-09-25 21:57:40.716 UTC [13091] LOG: received smart shutdown request
2026-09-25 21:57:40.719 UTC [13091] LOG: background worker "logical replication launcher" (PID 13100) exited with exit code 1
2026-09-25 21:57:40.719 UTC [13095] LOG: shutting down
2026-09-25 21:57:40.896 UTC [13091] LOG: database system is shut down
2026-09-25T21:57:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46696 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:57:41 dist-test-slave-2g0f krb5kdc[14073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:57:41 dist-test-slave-2g0f krb5kdc[14073](info): set up 2 sockets
Sep 25 21:57:41 dist-test-slave-2g0f krb5kdc[14073](info): commencing operation
krb5kdc: starting...
W20260925 21:57:41.931547 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:57:42 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:57:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.70.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.70.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/rangeradmin_127.2.70.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/rangeradmin_127.2.70.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.70.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.70.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/rangerlookup_127.2.70.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/rangerlookup_127.2.70.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.70.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.70.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/HTTP_127.2.70.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/HTTP_127.2.70.212@KRBTEST.COM.keytab.
I20260925 21:57:42.091987 2331 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdR78Ha/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/postgres -l logfile start
2026-09-25 21:57:44.624 UTC [14103] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 21:57:44.624 UTC [14103] LOG: listening on IPv4 address "127.2.70.212", port 58063
2026-09-25 21:57:44.628 UTC [14103] LOG: listening on Unix socket "/tmp/.s.PGSQL.58063"
2026-09-25 21:57:44.649 UTC [14106] LOG: database system was shut down at 2026-09-25 21:57:43 UTC
2026-09-25 21:57:44.653 UTC [14103] LOG: database system is ready to accept connections
I20260925 21:57:45.523411 2331 mini_postgres.cc:96] Postgres bound to 58063
2026-09-25 21:57:45.528 UTC [14114] FATAL: database "slave" does not exist
127.2.70.212:58063 - accepting connections
I20260925 21:57:45.529443 2331 mini_ranger.cc:161] Starting Ranger...
I20260925 21:57:45.543332 2331 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 21:57:46.053000 2331 mini_ranger.cc:87] Created ranger Postgres database
I20260925 21:57:46.053098 2331 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-admin
2026-09-25 21:57:46,325 [I] DB FLAVOR :POSTGRES
2026-09-25 21:57:46,325 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 21:57:46,326 [I] Checking connection..
2026-09-25 21:57:46,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 21:57:46,568 [I] Checking connection passed.
2026-09-25 21:57:46,568 [I] --------- Verifying version history table ---------
2026-09-25 21:57:46,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 21:57:46,802 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 21:57:46,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 21:57:47,042 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 21:57:47,042 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 21:57:47,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 21:57:47.272 UTC [14178] WARNING: there is no transaction in progress
2026-09-25 21:57:47,278 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 21:57:47,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 21:57:47,511 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 21:57:47,512 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 21:57:47,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 21:57:47,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 21:57:47,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2g0f.c.gcp-upstream.internal','N') ;"
2026-09-25 21:57:48,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 21:57:48,498 [I] Table x_portal_user does not exist in database ranger
2026-09-25 21:57:48,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 21:57:48,740 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 21:57:48,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 21:57:48,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 21:57:49,237 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 21:57:49,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 21:57:50.241 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.258 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.276 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.293 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.310 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.329 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.386 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.393 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.409 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.420 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.431 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.445 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.455 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.461 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.469 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:50.476 UTC [14286] WARNING: there is no transaction in progress
2026-09-25 21:57:51,180 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 21:57:51,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 21:57:51,420 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 21:57:51,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 21:57:51,675 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 21:57:51,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 21:57:51,937 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 21:57:51,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 21:57:52,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 21:57:52,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2g0f.c.gcp-upstream.internal';"
2026-09-25 21:57:52,650 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 21:57:52,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 21:57:52,894 [I] DB_PATCHES have already been applied
I20260925 21:57:52.900095 2331 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdR78Ha/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 21:57:52.900161 2331 mini_ranger.cc:193] Using host: 127.2.70.212
I20260925 21:57:52.902985 2331 mini_ranger.cc:227] Ranger admin URL: http://127.2.70.212:60693
Sep 25, 2026 9:57:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 9:57:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 9:57:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 9:57:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 9:57:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 9:57:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 9:57:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 21:57:53.897014 2331 mini_ranger.cc:160] Time spent starting Ranger: real 8.368s user 0.000s sys 0.008s
Sep 25, 2026 9:58:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 9:58:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 9:58:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 9:58:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 9:58:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 9:58:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 21:58:16.188651 2331 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.70.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.70.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/rangerkms_127.2.70.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.70.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/rangerkms_127.2.70.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.70.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.70.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.70.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/HTTP_127.2.70.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.70.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/HTTP_127.2.70.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 21:58:16 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373496, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 21:58:16.285595 2331 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 21:58:16.295019 2331 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 21:58:17.228029 2331 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 21:58:17.228103 2331 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-kms
I20260925 21:58:17.228118 2331 mini_ranger_kms.cc:227] Using postgres at 127.2.70.212:58063
2026-09-25 21:58:17,373 [I] DB FLAVOR :POSTGRES
2026-09-25 21:58:17,373 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 21:58:17,373 [I] Checking connection
2026-09-25 21:58:17,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 21:58:17,601 [I] connection success
2026-09-25 21:58:17,602 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 21:58:17,602 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 21:58:17,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 21:58:17,850 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 21:58:17,850 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 21:58:17,851 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 21:58:17,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.212:58063/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 21:58:18,101 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 21:58:18.595444 2331 mini_ranger_kms.cc:310] Created kms service
I20260925 21:58:18.796339 2331 mini_ranger_kms.cc:326] Created kudu user
I20260925 21:58:18.839313 2331 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 21:58:18 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373496, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:58:19.231796 2331 mini_ranger_kms.cc:384] Added ranger policy
I20260925 21:58:19.232066 2331 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdR78Ha/build/release/bin/postgresql.jar:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdR78Ha/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 21:58:19.232089 2331 mini_ranger_kms.cc:240] Using host: 127.2.70.212
I20260925 21:58:19.235184 2331 mini_ranger_kms.cc:276] Ranger KMS PID: 14466
I20260925 21:58:19.235256 2331 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.70.212:36893
Sep 25, 2026 9:58:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 9:58:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 9:58:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 9:58:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 9:58:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 9:58:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 21:58:21.227169 2331 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.942s user 0.013s sys 0.117s
I20260925 21:58:21.227268 2331 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 21:58:21.227313 2331 mini_ranger_kms.cc:401] 127.2.70.212:36893/kms/v1/keys
Sep 25 21:58:21 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790373501, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.70.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 21:58:21 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373501, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.70.212@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
Sep 25 21:58:22 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373501, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.70.212@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
Sep 25 21:58:24 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:58:24.403728 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:33101
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:33101
--ranger_config_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.212:36893/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:24.506327 14517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:24.506541 14517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:24.506562 14517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:24.507728 14517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:58:24.507776 14517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:24.507788 14517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:24.507805 14517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:58:24.507817 14517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:58:24.509445 14517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.70.212:36893/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:33101
--ranger_config_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:33101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:24.509677 14517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:24.509895 14517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:58:24.516089 14517 server_base.cc:1034] running on GCE node
W20260925 21:58:24.516149 14525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:24.516152 14523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:24.516278 14522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:24.516628 14517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:24.516920 14517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:24.518069 14517 hybrid_clock.cc:648] HybridClock initialized: now 1790373504518044 us; error 37 us; skew 500 ppm
Sep 25 21:58:24 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373504, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:24.524619 14517 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:58:24.525205 14517 webserver.cc:460] Webserver started at http://127.2.70.254:41793/ using document root <none> and password file <none>
I20260925 21:58:24.525465 14517 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:24.525508 14517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:24.525653 14517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 21:58:24 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373504, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:58:24.581583 14517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "3b4c5421e6f048e683329eaa9b4ba5d5"
format_stamp: "Formatted at 2026-09-25 21:58:24 on dist-test-slave-2g0f"
server_key: "ad079b75f6a12df9b73fafdf226255c0"
server_key_iv: "c1682cee485a7b4201c8b664a2d3fa19"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:24.582036 14517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "3b4c5421e6f048e683329eaa9b4ba5d5"
format_stamp: "Formatted at 2026-09-25 21:58:24 on dist-test-slave-2g0f"
server_key: "ad079b75f6a12df9b73fafdf226255c0"
server_key_iv: "c1682cee485a7b4201c8b664a2d3fa19"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:24.583806 14517 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 21:58:24.596679 14532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:24.596982 14517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:24.597105 14517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "3b4c5421e6f048e683329eaa9b4ba5d5"
format_stamp: "Formatted at 2026-09-25 21:58:24 on dist-test-slave-2g0f"
server_key: "ad079b75f6a12df9b73fafdf226255c0"
server_key_iv: "c1682cee485a7b4201c8b664a2d3fa19"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:24.597208 14517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:24.631738 14517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:24.634044 14517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:24.634198 14517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:24.639170 14517 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:33101
I20260925 21:58:24.639200 14584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:33101 every 8 connection(s)
I20260925 21:58:24.639679 14517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:58:24.640370 14585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:24.641436 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 14517
I20260925 21:58:24.641534 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:58:24.641709 2331 external_mini_cluster.cc:1428] Setting key 872db15fdc8b07d39d1585f508487fea
I20260925 21:58:24.643276 14585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5: Bootstrap starting.
I20260925 21:58:24.643925 14585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:24.644232 14585 log.cc:824] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:24.644986 14585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5: No bootstrap required, opened a new log
I20260925 21:58:24.647058 14585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33101 } }
I20260925 21:58:24.647243 14585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:24.647303 14585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4c5421e6f048e683329eaa9b4ba5d5, State: Initialized, Role: FOLLOWER
I20260925 21:58:24.647480 14585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33101 } }
I20260925 21:58:24.647550 14585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:58:24.647580 14585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:58:24.647640 14585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:24.648245 14585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33101 } }
I20260925 21:58:24.648344 14585 leader_election.cc:302] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b4c5421e6f048e683329eaa9b4ba5d5; no voters:
I20260925 21:58:24.648523 14585 leader_election.cc:288] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:58:24.648567 14590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:58:24.648774 14590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3b4c5421e6f048e683329eaa9b4ba5d5, State: Running, Role: LEADER
I20260925 21:58:24.648877 14585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:58:24.648944 14590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33101 } }
I20260925 21:58:24.649322 14590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33101 } } }
Sep 25 21:58:24 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:24.649475 14590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:24.649592 14590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b4c5421e6f048e683329eaa9b4ba5d5. Latest consensus state: current_term: 1 leader_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33101 } } }
I20260925 21:58:24.649710 14590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:24.649864 14599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:58:24.650413 14599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:58:24.650691 14585 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 21:58:24.651726 14607 mvcc.cc:205] Tried to move back new op lower bound from 7333369875051773952 to 7333369875042607104. Current Snapshot: MvccSnapshot[applied={T|T < 7333369875051773952}]
I20260925 21:58:24.652221 14599 catalog_manager.cc:1347] Generated new cluster ID: 1ee84a75db21460b8638f4f7657a62e4
I20260925 21:58:24.652262 14599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:58:24.656693 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:24.642228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.2.70.254:33101)
0925 21:58:24.642407 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:24.642416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:24.642449 (+ 33us) server_negotiation.cc:407] Connection header received
0925 21:58:24.642984 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:24.642993 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:24.643064 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:24.643216 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:24.643967 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:24.645216 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:24.646374 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:24.647010 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:24.647132 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:24.650074 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:24.650078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:24.650079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:24.650092 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:24.652014 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:24.652403 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:24.652406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:24.652406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:24.652511 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:24.652689 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:24.652690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:24.652690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:24.654739 (+ 2049us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:24.655092 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:24.655680 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 21:58:24.656303 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260925 21:58:24.665030 14599 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:58:24.665879 14599 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:58:24.671540 14599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5: Generated new TSK 0
I20260925 21:58:24.671727 14599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:58:25.373006 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.212:36893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:33101
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:25.492370 14630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:25.492667 14630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:25.492705 14630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:25.494452 14630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:25.494514 14630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:25.494553 14630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:58:25.496898 14630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.70.212:36893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:33101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:25.497164 14630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:25.497442 14630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:25.502684 14636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:25.502673 14638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:25.502698 14635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:25.502807 14630 server_base.cc:1034] running on GCE node
I20260925 21:58:25.503304 14630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:25.503526 14630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:25.504657 14630 hybrid_clock.cc:648] HybridClock initialized: now 1790373505504642 us; error 30 us; skew 500 ppm
Sep 25 21:58:25 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373505, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:25.507166 14630 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:58:25.507614 14630 webserver.cc:460] Webserver started at http://127.2.70.193:44097/ using document root <none> and password file <none>
I20260925 21:58:25.507870 14630 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:25.507938 14630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:25.508075 14630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 21:58:25 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373505, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:58:25.522598 14630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "bc212218a4234f13a15d533ede7cced5"
format_stamp: "Formatted at 2026-09-25 21:58:25 on dist-test-slave-2g0f"
server_key: "02435af01a7482278e15853a1430c05b"
server_key_iv: "ef5a44d250122febb0cd8b679ff89255"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:25.523185 14630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "bc212218a4234f13a15d533ede7cced5"
format_stamp: "Formatted at 2026-09-25 21:58:25 on dist-test-slave-2g0f"
server_key: "02435af01a7482278e15853a1430c05b"
server_key_iv: "ef5a44d250122febb0cd8b679ff89255"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:25.525154 14630 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 21:58:25.539867 14645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:25.540145 14630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:58:25.540256 14630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "bc212218a4234f13a15d533ede7cced5"
format_stamp: "Formatted at 2026-09-25 21:58:25 on dist-test-slave-2g0f"
server_key: "02435af01a7482278e15853a1430c05b"
server_key_iv: "ef5a44d250122febb0cd8b679ff89255"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:25.540354 14630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:25.570463 14630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:25.572772 14630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:25.572921 14630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:25.573164 14630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:25.573525 14630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:25.573563 14630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:25.573592 14630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:25.573668 14630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:25.580288 14630 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:38313
I20260925 21:58:25.580653 14630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:58:25.581200 14760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:38313 every 8 connection(s)
Sep 25 21:58:25 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373505, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:25.590103 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 14630
I20260925 21:58:25.590209 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:58:25.590394 2331 external_mini_cluster.cc:1428] Setting key 286970da305ea80da43faf103e1aea71
I20260925 21:58:25.592439 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:25.581470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:33101 (local address 127.2.70.193:60515)
0925 21:58:25.581932 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:25.581984 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:25.582646 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:25.582955 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:25.582961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:25.583217 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:25.583520 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:25.583527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:25.584290 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:25.584292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:25.585556 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:25.585559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:25.586086 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:25.586087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:25.586230 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:25.586894 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:25.586904 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:25.588468 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:25.590630 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:25.590638 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:25.590641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:25.590966 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:25.591186 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:25.591187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:25.591187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:25.591327 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:25.591653 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:25.591659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:25.591882 (+ 223us) client_negotiation.cc:769] Sending connection context
0925 21:58:25.592002 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 21:58:25.592180 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":91,"threads_started":1}
I20260925 21:58:25.592967 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:25.581468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:60515 (local address 127.2.70.254:33101)
0925 21:58:25.581621 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:25.581623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:25.582075 (+ 452us) server_negotiation.cc:407] Connection header received
0925 21:58:25.582758 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:25.582759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:25.582770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:25.582847 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:25.583620 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:25.584198 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:25.585659 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:25.585975 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:25.586059 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:25.588584 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:25.588586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:25.588586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:25.588593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:25.590514 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:25.591053 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:25.591054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:25.591055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:25.591122 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:25.591402 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:25.591417 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:25.591417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:25.591562 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:25.591623 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:25.592721 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0925 21:58:25.592866 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260925 21:58:25.593441 14761 heartbeater.cc:347] Connected to a master server at 127.2.70.254:33101
I20260925 21:58:25.593537 14761 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:25.593730 14761 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
I20260925 21:58:25.594354 14549 ts_manager.cc:194] Registered new tserver with Master: bc212218a4234f13a15d533ede7cced5 (127.2.70.193:38313)
I20260925 21:58:25.595278 14549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:60515
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:58:25.619381 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.212:36893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:33101
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 21:58:25 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790373505, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 21:58:25.738238 14767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:25.738570 14767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:25.738658 14767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:25.740715 14767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:25.740833 14767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:25.740927 14767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:58:25.743156 14767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.70.212:36893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:33101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:25.743624 14767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:25.743992 14767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:25.751304 14774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:25.751308 14775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:25.751381 14777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:25.752624 14767 server_base.cc:1034] running on GCE node
I20260925 21:58:25.752839 14767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:25.753130 14767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:25.756395 14767 hybrid_clock.cc:648] HybridClock initialized: now 1790373505756367 us; error 40 us; skew 500 ppm
Sep 25 21:58:25 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373505, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:25.759672 14767 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:58:25.760226 14767 webserver.cc:460] Webserver started at http://127.2.70.194:36315/ using document root <none> and password file <none>
I20260925 21:58:25.760489 14767 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:25.760545 14767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:25.760708 14767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 21:58:25 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373505, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:58:25.778434 14767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "eb4cae636df24731b0b14518d325fec0"
format_stamp: "Formatted at 2026-09-25 21:58:25 on dist-test-slave-2g0f"
server_key: "9c4f300dcdee24e8c7513f16d032ebd1"
server_key_iv: "9422a15b4a88bd4545ee8325883c713c"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:25.778923 14767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "eb4cae636df24731b0b14518d325fec0"
format_stamp: "Formatted at 2026-09-25 21:58:25 on dist-test-slave-2g0f"
server_key: "9c4f300dcdee24e8c7513f16d032ebd1"
server_key_iv: "9422a15b4a88bd4545ee8325883c713c"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:25.781477 14767 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 21:58:25.797602 14785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:25.797850 14767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:58:25.797930 14767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "eb4cae636df24731b0b14518d325fec0"
format_stamp: "Formatted at 2026-09-25 21:58:25 on dist-test-slave-2g0f"
server_key: "9c4f300dcdee24e8c7513f16d032ebd1"
server_key_iv: "9422a15b4a88bd4545ee8325883c713c"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:25.798017 14767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:25.833058 14767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:25.835467 14767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:25.835630 14767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:25.835906 14767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:25.836292 14767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:25.836346 14767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:25.836427 14767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:25.836458 14767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:25.842728 14767 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:33859
I20260925 21:58:25.843255 14767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:58:25.849685 14898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:33859 every 8 connection(s)
I20260925 21:58:25.850217 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 14767
I20260925 21:58:25.850311 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:58:25.850502 2331 external_mini_cluster.cc:1428] Setting key b6651a27e7c40ec2ed7b153cfa18c1fb
Sep 25 21:58:25 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373505, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:25.862746 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:25.850777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:43667 (local address 127.2.70.254:33101)
0925 21:58:25.850910 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:25.850913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:25.851161 (+ 248us) server_negotiation.cc:407] Connection header received
0925 21:58:25.851874 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:25.851875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:25.851973 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:25.852013 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:25.853128 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:25.853874 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:25.855367 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:25.855687 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:25.855728 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:25.859331 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:25.859333 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:25.859333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:25.859342 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:25.861055 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:25.861568 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:25.861570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:25.861570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:25.861635 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:25.861954 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:25.861955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:25.861956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:25.862104 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:25.862151 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:25.862550 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 21:58:25.862647 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 21:58:25.863550 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:25.850445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:33101 (local address 127.2.70.194:43667)
0925 21:58:25.851022 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:25.851072 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:25.851721 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:25.852422 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:25.852426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:25.852668 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:25.853011 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:25.853018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:25.854007 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:25.854007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:25.855264 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:25.855267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:25.855817 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:25.855818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:25.856016 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:25.857490 (+ 1474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:25.857502 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:25.859210 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:25.861156 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:25.861163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:25.861166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:25.861473 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:25.861709 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:25.861710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:25.861711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:25.861855 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:25.862198 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:25.862205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:25.862382 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 21:58:25.863131 (+ 749us) client_negotiation.cc:240] Negotiation successful
0925 21:58:25.863308 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":103,"threads_started":1}
I20260925 21:58:25.864271 14899 heartbeater.cc:347] Connected to a master server at 127.2.70.254:33101
I20260925 21:58:25.864372 14899 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:25.864576 14899 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
I20260925 21:58:25.864975 14549 ts_manager.cc:194] Registered new tserver with Master: eb4cae636df24731b0b14518d325fec0 (127.2.70.194:33859)
I20260925 21:58:25.865465 14549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:43667
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:58:25.912125 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.212:36893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:33101
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:26.042012 14904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:26.042377 14904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:26.042464 14904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:26.044269 14904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:26.044389 14904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:26.044473 14904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:58:26.046715 14904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.70.212:36893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:33101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:26.047201 14904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:26.047536 14904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:26.054316 14910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:26.054705 14909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:26.054937 14912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:26.055797 14904 server_base.cc:1034] running on GCE node
I20260925 21:58:26.056141 14904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:26.056422 14904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:26.061267 14904 hybrid_clock.cc:648] HybridClock initialized: now 1790373506061237 us; error 35 us; skew 500 ppm
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:26.064538 14904 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:58:26.064994 14904 webserver.cc:460] Webserver started at http://127.2.70.195:34039/ using document root <none> and password file <none>
I20260925 21:58:26.065227 14904 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:26.065279 14904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:26.065409 14904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:58:26.081888 14904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b"
format_stamp: "Formatted at 2026-09-25 21:58:26 on dist-test-slave-2g0f"
server_key: "ecc5f9a6fe0e0e52c9b5d8739cd8369c"
server_key_iv: "2a08952fb2fd2396bf50e079e433643e"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:26.082422 14904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b"
format_stamp: "Formatted at 2026-09-25 21:58:26 on dist-test-slave-2g0f"
server_key: "ecc5f9a6fe0e0e52c9b5d8739cd8369c"
server_key_iv: "2a08952fb2fd2396bf50e079e433643e"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:26.085029 14904 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 21:58:26.112001 14919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.112293 14904 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260925 21:58:26.112448 14904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b"
format_stamp: "Formatted at 2026-09-25 21:58:26 on dist-test-slave-2g0f"
server_key: "ecc5f9a6fe0e0e52c9b5d8739cd8369c"
server_key_iv: "2a08952fb2fd2396bf50e079e433643e"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:26.112632 14904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:26.128234 14904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:26.130857 14904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:26.131088 14904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:26.131371 14904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:26.132050 14904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:26.132160 14904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.132244 14904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:26.132308 14904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.139292 14904 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:46795
I20260925 21:58:26.142624 15032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:46795 every 8 connection(s)
I20260925 21:58:26.143059 14904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:58:26.149708 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 14904
I20260925 21:58:26.149834 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:58:26.150014 2331 external_mini_cluster.cc:1428] Setting key c6efd38cd4242478e39ff259b6f21cb6
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:26.156600 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:26.144604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:32967 (local address 127.2.70.254:33101)
0925 21:58:26.144654 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:26.144657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:26.144877 (+ 220us) server_negotiation.cc:407] Connection header received
0925 21:58:26.145639 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:26.145641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:26.145657 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:26.145693 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:26.146604 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.147352 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.148929 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.149225 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.149277 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:26.152981 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:26.152983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:26.152983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:26.152992 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:26.154812 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:26.155361 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:26.155363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:26.155363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:26.155437 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:26.155747 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:26.155748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:26.155748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:26.155937 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:26.155986 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:26.156396 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 21:58:26.156496 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 21:58:26.157460 15035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:26.144276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:33101 (local address 127.2.70.195:32967)
0925 21:58:26.144736 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:26.144787 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:26.145498 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:26.145901 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:26.145905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:26.146180 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:26.146482 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:26.146488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.147485 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.147486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:26.148820 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:26.148824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.149411 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.149412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:26.149600 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:26.150400 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:26.150978 (+ 578us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:26.152846 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:26.154928 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:26.154936 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:26.154939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:26.155252 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:26.155517 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:26.155518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:26.155518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:26.155662 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:26.156038 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:26.156045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:26.156220 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 21:58:26.157002 (+ 782us) client_negotiation.cc:240] Negotiation successful
0925 21:58:26.157195 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":99,"threads_started":1}
I20260925 21:58:26.158200 15033 heartbeater.cc:347] Connected to a master server at 127.2.70.254:33101
I20260925 21:58:26.158303 15033 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:26.158522 15033 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
I20260925 21:58:26.159006 14549 ts_manager.cc:194] Registered new tserver with Master: c4ac4bbb01684f6c8c8e85a264ebf16b (127.2.70.195:46795)
I20260925 21:58:26.159502 14549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:32967
I20260925 21:58:26.161769 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:58:26.162410 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 14630
I20260925 21:58:26.168807 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 14767
I20260925 21:58:26.173988 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 14904
I20260925 21:58:26.180814 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 14517
I20260925 21:58:26.187076 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:33101
--webserver_interface=127.2.70.254
--webserver_port=41793
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:33101
--ranger_config_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.212:36893/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.70.212:36893/kms with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:26.267653 15036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:26.267887 15036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:26.267925 15036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:26.269129 15036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:58:26.269174 15036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:26.269187 15036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:26.269199 15036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:58:26.269210 15036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:58:26.270845 15036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.70.212:36893/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:33101
--ranger_config_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:33101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=41793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:26.271087 15036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:26.271311 15036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:26.275993 15041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:26.276024 15036 server_base.cc:1034] running on GCE node
W20260925 21:58:26.276163 15042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:26.276007 15044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:26.276414 15036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:26.276703 15036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:26.277872 15036 hybrid_clock.cc:648] HybridClock initialized: now 1790373506277858 us; error 26 us; skew 500 ppm
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:26.280344 15036 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:58:26.280797 15036 webserver.cc:460] Webserver started at http://127.2.70.254:41793/ using document root <none> and password file <none>
I20260925 21:58:26.281021 15036 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:26.281071 15036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:58:26.322654 15036 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 21:58:26.323329 15052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.323542 15036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:26.323660 15036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "3b4c5421e6f048e683329eaa9b4ba5d5"
format_stamp: "Formatted at 2026-09-25 21:58:24 on dist-test-slave-2g0f"
server_key: "ad079b75f6a12df9b73fafdf226255c0"
server_key_iv: "c1682cee485a7b4201c8b664a2d3fa19"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:26.324154 15036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:26.342139 15036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:26.343648 15036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:26.343758 15036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:26.347590 15036 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:33101
I20260925 21:58:26.347661 15104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:33101 every 8 connection(s)
I20260925 21:58:26.348004 15036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:58:26.349225 15105 sys_catalog.cc:263] Verifying existing consensus state
I20260925 21:58:26.349853 15105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5: Bootstrap starting.
I20260925 21:58:26.352598 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 15036
I20260925 21:58:26.352861 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.212:36893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:38313
--local_ip_for_outbound_sockets=127.2.70.193
--tserver_master_addrs=127.2.70.254:33101
--webserver_port=44097
--webserver_interface=127.2.70.193
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.70.212:36893/kms with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:58:26.357798 15105 log.cc:824] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:26.359437 15105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 21:58:26.359694 15105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5: Bootstrap complete.
I20260925 21:58:26.361900 15105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33101 } }
I20260925 21:58:26.362138 15105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4c5421e6f048e683329eaa9b4ba5d5, State: Initialized, Role: FOLLOWER
I20260925 21:58:26.362295 15105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33101 } }
I20260925 21:58:26.362366 15105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:58:26.362416 15105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:58:26.362460 15105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 21:58:26.363433 15105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33101 } }
I20260925 21:58:26.363529 15105 leader_election.cc:302] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b4c5421e6f048e683329eaa9b4ba5d5; no voters:
I20260925 21:58:26.363698 15105 leader_election.cc:288] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 21:58:26.363744 15109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 21:58:26.363916 15109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [term 2 LEADER]: Becoming Leader. State: Replica: 3b4c5421e6f048e683329eaa9b4ba5d5, State: Running, Role: LEADER
I20260925 21:58:26.363978 15105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:58:26.364038 15109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33101 } }
I20260925 21:58:26.364357 15109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b4c5421e6f048e683329eaa9b4ba5d5. Latest consensus state: current_term: 2 leader_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33101 } } }
I20260925 21:58:26.364396 15110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c5421e6f048e683329eaa9b4ba5d5" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33101 } } }
I20260925 21:58:26.364516 15109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:26.364540 15110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:26.364734 15114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:58:26.365195 15114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:58:26.365401 15114 catalog_manager.cc:1259] Loaded cluster ID: 1ee84a75db21460b8638f4f7657a62e4
I20260925 21:58:26.365471 15114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:58:26.365561 15105 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 21:58:26.366751 15114 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:58:26.366932 15114 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5: Loaded TSK: 0
I20260925 21:58:26.367146 15114 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 21:58:26.462786 15107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:26.463073 15107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:26.463112 15107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:26.465039 15107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:26.465101 15107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:26.465142 15107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:58:26.467429 15107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.70.212:36893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:38313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=44097
--tserver_master_addrs=127.2.70.254:33101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:26.467953 15107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:26.468236 15107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:26.474886 15143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:26.474978 15145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:26.475271 15107 server_base.cc:1034] running on GCE node
W20260925 21:58:26.475533 15142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:26.475823 15107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:26.476137 15107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:26.477310 15107 hybrid_clock.cc:648] HybridClock initialized: now 1790373506477279 us; error 45 us; skew 500 ppm
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:26.480495 15107 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:58:26.481081 15107 webserver.cc:460] Webserver started at http://127.2.70.193:44097/ using document root <none> and password file <none>
I20260925 21:58:26.481344 15107 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:26.481431 15107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:58:26.504354 15107 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260925 21:58:26.505541 15153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.505710 15107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:26.505797 15107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "bc212218a4234f13a15d533ede7cced5"
format_stamp: "Formatted at 2026-09-25 21:58:25 on dist-test-slave-2g0f"
server_key: "02435af01a7482278e15853a1430c05b"
server_key_iv: "ef5a44d250122febb0cd8b679ff89255"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:26.506302 15107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:26.528019 15107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:26.530656 15107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:26.530876 15107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:26.531193 15107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:26.531620 15107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:26.531704 15107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.531749 15107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:26.531767 15107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.538241 15107 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:38313
I20260925 21:58:26.538798 15107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:58:26.539135 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 15107
I20260925 21:58:26.539404 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.212:36893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:33859
--local_ip_for_outbound_sockets=127.2.70.194
--tserver_master_addrs=127.2.70.254:33101
--webserver_port=36315
--webserver_interface=127.2.70.194
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.70.212:36893/kms with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:58:26.540736 15266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:38313 every 8 connection(s)
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:26.563745 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:26.542481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:48475 (local address 127.2.70.254:33101)
0925 21:58:26.545190 (+ 2709us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:26.545201 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:26.545246 (+ 45us) server_negotiation.cc:407] Connection header received
0925 21:58:26.545460 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:26.545469 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:26.545537 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:26.545681 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:26.546539 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.547966 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.549489 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.550123 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.550247 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:26.559576 (+ 9329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:26.559583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:26.559583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:26.559597 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:26.561574 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:26.562182 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:26.562186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:26.562186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:26.562316 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:26.562586 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:26.562587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:26.562587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:26.562897 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:26.562954 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:26.563356 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 21:58:26.563504 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2651,"thread_start_us":75,"threads_started":1}
I20260925 21:58:26.564622 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:26.542127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:33101 (local address 127.2.70.193:48475)
0925 21:58:26.542318 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:26.542370 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:26.543012 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:26.545869 (+ 2857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:26.545873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:26.546126 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:26.546398 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:26.546404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.548100 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.548101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:26.549379 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:26.549383 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.550260 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.550261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:26.557067 (+ 6806us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:26.557701 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:26.557713 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:26.559404 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:26.561695 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:26.561704 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:26.561706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:26.562022 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:26.562386 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:26.562387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:26.562387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:26.562519 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:26.562997 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:26.563003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:26.563165 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 21:58:26.564221 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0925 21:58:26.564400 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260925 21:58:26.565737 15267 heartbeater.cc:347] Connected to a master server at 127.2.70.254:33101
I20260925 21:58:26.565830 15267 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:26.566013 15267 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
W20260925 21:58:26.566887 15267 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:33101 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 21:58:26.566955 15267 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:26.567018 15267 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
I20260925 21:58:26.567416 15267 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:26.567489 15267 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
W20260925 21:58:26.567793 15267 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 21:58:26.650539 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:26.650915 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:26.650980 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:26.652697 15269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:26.652753 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:26.652792 15269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:58:26.654340 15269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.70.212:36893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:33859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=36315
--tserver_master_addrs=127.2.70.254:33101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:26.654592 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:26.654834 15269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:26.659669 15276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:26.659716 15269 server_base.cc:1034] running on GCE node
W20260925 21:58:26.659802 15277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:26.659682 15279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:26.660204 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:26.660475 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:26.661634 15269 hybrid_clock.cc:648] HybridClock initialized: now 1790373506661620 us; error 33 us; skew 500 ppm
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:26.664731 15269 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:58:26.665400 15269 webserver.cc:460] Webserver started at http://127.2.70.194:36315/ using document root <none> and password file <none>
I20260925 21:58:26.665670 15269 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:26.665748 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:58:26.685596 15269 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:26.686666 15287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.686885 15269 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:26.686998 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "eb4cae636df24731b0b14518d325fec0"
format_stamp: "Formatted at 2026-09-25 21:58:25 on dist-test-slave-2g0f"
server_key: "9c4f300dcdee24e8c7513f16d032ebd1"
server_key_iv: "9422a15b4a88bd4545ee8325883c713c"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:26.687479 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:26.703668 15269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:26.706266 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:26.706499 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:26.706784 15269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:26.707191 15269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:26.707243 15269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.707278 15269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:26.707302 15269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.714614 15269 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:33859
I20260925 21:58:26.714695 15400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:33859 every 8 connection(s)
I20260925 21:58:26.715170 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:58:26.721036 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 15269
I20260925 21:58:26.721362 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.212:36893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:46795
--local_ip_for_outbound_sockets=127.2.70.195
--tserver_master_addrs=127.2.70.254:33101
--webserver_port=34039
--webserver_interface=127.2.70.195
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.70.212:36893/kms with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:26.728015 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:26.715908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:55317 (local address 127.2.70.254:33101)
0925 21:58:26.716495 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:26.716497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:26.716508 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:58:26.716858 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:26.716859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:26.716872 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:26.716914 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:26.717697 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.718187 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.719656 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.719952 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.720019 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:26.724516 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:26.724518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:26.724518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:26.724526 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:26.726286 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:26.726787 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:26.726789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:26.726790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:26.726861 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:26.727165 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:26.727165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:26.727166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:26.727317 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:26.727389 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:26.727762 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 21:58:26.727881 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562}
I20260925 21:58:26.728361 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:26.715912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:33101 (local address 127.2.70.194:55317)
0925 21:58:26.716095 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:26.716144 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:26.716739 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:26.717085 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:26.717090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:26.717324 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:26.717607 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:26.717614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.718276 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.718277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:26.719559 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:26.719562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.720251 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.720252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:26.720360 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:26.722748 (+ 2388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:26.722759 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:26.724394 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:26.726387 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:26.726394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:26.726397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:26.726703 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:26.726943 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:26.726944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:26.726944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:26.727097 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:26.727384 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:26.727389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:26.727562 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 21:58:26.727733 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 21:58:26.727937 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260925 21:58:26.728881 15401 heartbeater.cc:347] Connected to a master server at 127.2.70.254:33101
I20260925 21:58:26.729034 15401 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:26.729305 15401 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
W20260925 21:58:26.730044 15401 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:33101 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 21:58:26.730185 15401 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:26.730290 15401 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
I20260925 21:58:26.730707 15401 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:26.730814 15401 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
W20260925 21:58:26.731204 15401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 21:58:26.821687 15404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:26.821904 15404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:26.821938 15404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:26.823136 15404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:26.823179 15404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:26.823216 15404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:58:26.824795 15404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.70.212:36893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:46795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=34039
--tserver_master_addrs=127.2.70.254:33101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:26.825032 15404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:26.825285 15404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:26.831727 15413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:26.831732 15411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:26.832418 15404 server_base.cc:1034] running on GCE node
W20260925 21:58:26.835283 15410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:26.835955 15404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:26.836292 15404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:26.837474 15404 hybrid_clock.cc:648] HybridClock initialized: now 1790373506837464 us; error 33 us; skew 500 ppm
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:26.840541 15404 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:58:26.840986 15404 webserver.cc:460] Webserver started at http://127.2.70.195:34039/ using document root <none> and password file <none>
I20260925 21:58:26.841197 15404 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:26.841260 15404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:58:26.862828 15404 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260925 21:58:26.863981 15421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.864215 15404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:26.864316 15404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b"
format_stamp: "Formatted at 2026-09-25 21:58:26 on dist-test-slave-2g0f"
server_key: "ecc5f9a6fe0e0e52c9b5d8739cd8369c"
server_key_iv: "2a08952fb2fd2396bf50e079e433643e"
server_key_version: "kuduclusterkey@0"
I20260925 21:58:26.864862 15404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:26.917527 15404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:26.919921 15404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:26.920137 15404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:26.920445 15404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:26.920852 15404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:26.920941 15404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.921000 15404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:26.921041 15404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:26.927139 15404 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:46795
I20260925 21:58:26.927309 15534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:46795 every 8 connection(s)
I20260925 21:58:26.927592 15404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:58:26.933123 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 15404
Sep 25 21:58:26 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:26.941006 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:26.928469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:55983 (local address 127.2.70.254:33101)
0925 21:58:26.928536 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:26.928538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:26.932161 (+ 3623us) server_negotiation.cc:407] Connection header received
0925 21:58:26.932694 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:26.932696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:26.932710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:26.932762 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:26.933538 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.934071 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.935182 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.935446 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.935502 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:26.937822 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:26.937823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:26.937824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:26.937831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:26.939407 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:26.939926 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:26.939927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:26.939928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:26.939988 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:26.940272 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:26.940273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:26.940274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:26.940420 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:26.940472 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:26.940829 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 21:58:26.940910 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 21:58:26.941816 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:26.928403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:33101 (local address 127.2.70.195:55983)
0925 21:58:26.931978 (+ 3575us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:26.932031 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:26.932582 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:26.932835 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:26.932857 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:26.933110 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:26.933430 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:26.933438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.934277 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.934278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:26.935090 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:26.935093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.935543 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.935544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:26.935690 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:26.936273 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:26.936284 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:26.937718 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:26.939486 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:26.939493 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:26.939496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:26.939800 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:26.940059 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:26.940060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:26.940060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:26.940196 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:26.940505 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:26.940512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:26.940684 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 21:58:26.941402 (+ 718us) client_negotiation.cc:240] Negotiation successful
0925 21:58:26.941573 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3502,"thread_start_us":75,"threads_started":1}
I20260925 21:58:26.942526 15535 heartbeater.cc:347] Connected to a master server at 127.2.70.254:33101
I20260925 21:58:26.942678 15535 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:26.942929 15535 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
W20260925 21:58:26.943558 15535 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:33101 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 21:58:26.943676 15535 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:26.943778 15535 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
I20260925 21:58:26.943959 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:26.934139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.2.70.254:33101)
0925 21:58:26.934261 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:26.934263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:26.935547 (+ 1284us) server_negotiation.cc:407] Connection header received
0925 21:58:26.935567 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:26.935567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:26.935579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:26.935634 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:26.936666 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.937957 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.939032 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:26.939334 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:26.939393 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:26.940798 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:26.940799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:26.940799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:26.940806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:26.942380 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:26.942737 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:26.942738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:26.942738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:26.942795 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:26.942999 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:26.942999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:26.943000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:26.943131 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:26.943292 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:26.943723 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 21:58:26.943805 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260925 21:58:26.944115 15535 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:26.944192 15535 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
W20260925 21:58:26.944456 15535 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 21:58:27 dist-test-slave-2g0f krb5kdc[14073](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790373507, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:27.568004 15267 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:27.568197 15267 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
I20260925 21:58:27.568673 15068 ts_manager.cc:194] Registered new tserver with Master: bc212218a4234f13a15d533ede7cced5 (127.2.70.193:38313)
I20260925 21:58:27.569680 15068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:48475
I20260925 21:58:27.731473 15401 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:27.731647 15401 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
I20260925 21:58:27.732388 15068 ts_manager.cc:194] Registered new tserver with Master: eb4cae636df24731b0b14518d325fec0 (127.2.70.194:33859)
I20260925 21:58:27.732926 15068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:55317
Sep 25 21:58:27 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373507, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:58:27.944646 15535 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:27.944800 15535 heartbeater.cc:511] Master 127.2.70.254:33101 requested a full tablet report, sending...
I20260925 21:58:27.945086 15068 ts_manager.cc:194] Registered new tserver with Master: c4ac4bbb01684f6c8c8e85a264ebf16b (127.2.70.195:46795)
I20260925 21:58:27.945559 15068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:55983
I20260925 21:58:27.954178 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:58:27.962467 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:27.955145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.2.70.254:33101)
0925 21:58:27.955249 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:27.955251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:27.955263 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:58:27.955330 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:27.955331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:27.955344 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:27.955431 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:27.956096 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.956883 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.957849 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.958166 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.958243 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:27.959599 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:27.959600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:27.959601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:27.959607 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:27.961318 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:27.961681 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:27.961682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:27.961682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:27.961747 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:27.961934 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:27.961935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:27.961936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:27.962069 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:27.962169 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:27.962270 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 21:58:27.962353 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
I20260925 21:58:27.964548 15068 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:58:27.967000 15068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:58:27.979617 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:27.974435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:38313 (local address 127.0.0.1:39594)
0925 21:58:27.974891 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:27.974898 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:27.974928 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:27.975250 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:27.975253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:27.975256 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:27.975327 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:27.975329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.976638 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.976639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:27.978218 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:27.978221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.979170 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.979171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:27.979250 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:27.979426 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 21:58:27.979470 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 21:58:27.979532 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":44,"thread_start_us":61,"threads_started":1}
I20260925 21:58:27.979748 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:27.974291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.2.70.195:46795)
0925 21:58:27.975199 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:27.975203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:27.975213 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:58:27.975247 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:27.975249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:27.975264 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:27.975511 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:27.975582 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.976655 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.978327 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.979104 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.979527 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:27.979600 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:27.979628 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 21:58:27.979670 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":61,"threads_started":1}
I20260925 21:58:27.980086 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:27.974187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:46795 (local address 127.0.0.1:40242)
0925 21:58:27.974706 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:27.974746 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:27.974803 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:27.975354 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:27.975355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:27.975357 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:27.975431 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:27.975433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.976835 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.976836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:27.978207 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:27.978210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.979340 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.979341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:27.979425 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:27.979430 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:58:27.979478 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 21:58:27.980023 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":692,"thread_start_us":85,"threads_started":1}
I20260925 21:58:27.980255 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:27.974523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.2.70.193:38313)
0925 21:58:27.974848 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:27.975038 (+ 190us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:27.975049 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:58:27.975086 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:27.975088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:27.975105 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:27.975395 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:27.975466 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.976542 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.978313 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.979068 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.980051 (+ 983us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:27.980120 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:27.980145 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 21:58:27.980186 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":49,"threads_started":1}
I20260925 21:58:27.981001 15469 tablet_service.cc:1467] Processing CreateTablet for tablet dab7529bc96a462d8d78a0dedcb12511 (DEFAULT_TABLE table=test-table [id=140a765878dc4446838a2122488414e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:58:27.980973 15201 tablet_service.cc:1467] Processing CreateTablet for tablet dab7529bc96a462d8d78a0dedcb12511 (DEFAULT_TABLE table=test-table [id=140a765878dc4446838a2122488414e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:58:27.981330 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab7529bc96a462d8d78a0dedcb12511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:27.981343 15469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab7529bc96a462d8d78a0dedcb12511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:27.982847 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:27.974567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:33859 (local address 127.0.0.1:35554)
0925 21:58:27.979894 (+ 5327us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:27.979904 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:27.979962 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:27.980234 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:27.980235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:27.980236 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:27.980277 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:27.980278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.981207 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.981208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:27.981940 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:27.981942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.982638 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.982639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:27.982696 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:27.982700 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:58:27.982745 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 21:58:27.982777 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5309,"thread_start_us":49,"threads_started":1}
I20260925 21:58:27.983162 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:27.974709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.2.70.194:33859)
0925 21:58:27.976782 (+ 2073us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:27.976786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:27.980022 (+ 3236us) server_negotiation.cc:407] Connection header received
0925 21:58:27.980061 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:27.980064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:27.980079 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:27.980145 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:27.980345 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.981121 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.982031 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:27.982563 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:27.982625 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:27.982678 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:27.983052 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 21:58:27.983093 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2039,"thread_start_us":51,"threads_started":1}
I20260925 21:58:27.983896 15335 tablet_service.cc:1467] Processing CreateTablet for tablet dab7529bc96a462d8d78a0dedcb12511 (DEFAULT_TABLE table=test-table [id=140a765878dc4446838a2122488414e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:58:27.984181 15335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab7529bc96a462d8d78a0dedcb12511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:27.984697 15568 tablet_bootstrap.cc:492] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5: Bootstrap starting.
I20260925 21:58:27.985662 15568 tablet_bootstrap.cc:654] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:27.986027 15568 log.cc:824] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:27.986405 15569 tablet_bootstrap.cc:492] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b: Bootstrap starting.
I20260925 21:58:27.987273 15568 tablet_bootstrap.cc:492] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5: No bootstrap required, opened a new log
I20260925 21:58:27.987532 15568 ts_tablet_manager.cc:1397] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 21:58:27.988008 15569 tablet_bootstrap.cc:654] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:27.988381 15569 log.cc:824] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:27.988765 15571 tablet_bootstrap.cc:492] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0: Bootstrap starting.
I20260925 21:58:27.989411 15571 tablet_bootstrap.cc:654] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:27.989748 15571 log.cc:824] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:27.990489 15571 tablet_bootstrap.cc:492] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0: No bootstrap required, opened a new log
I20260925 21:58:27.990549 15571 ts_tablet_manager.cc:1397] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:58:27.991264 15568 raft_consensus.cc:348] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46795 } } peers { permanent_uuid: "bc212218a4234f13a15d533ede7cced5" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38313 } } peers { permanent_uuid: "eb4cae636df24731b0b14518d325fec0" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33859 } }
I20260925 21:58:27.991421 15568 raft_consensus.cc:374] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:27.991448 15568 raft_consensus.cc:729] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc212218a4234f13a15d533ede7cced5, State: Initialized, Role: FOLLOWER
I20260925 21:58:27.991613 15568 consensus_queue.cc:260] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46795 } } peers { permanent_uuid: "bc212218a4234f13a15d533ede7cced5" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38313 } } peers { permanent_uuid: "eb4cae636df24731b0b14518d325fec0" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33859 } }
I20260925 21:58:27.991912 15568 ts_tablet_manager.cc:1428] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260925 21:58:27.992015 15267 heartbeater.cc:503] Master 127.2.70.254:33101 was elected leader, sending a full tablet report...
I20260925 21:58:27.992651 15571 raft_consensus.cc:348] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46795 } } peers { permanent_uuid: "bc212218a4234f13a15d533ede7cced5" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38313 } } peers { permanent_uuid: "eb4cae636df24731b0b14518d325fec0" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33859 } }
I20260925 21:58:27.992796 15571 raft_consensus.cc:374] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:27.992826 15571 raft_consensus.cc:729] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4cae636df24731b0b14518d325fec0, State: Initialized, Role: FOLLOWER
I20260925 21:58:27.992933 15571 consensus_queue.cc:260] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46795 } } peers { permanent_uuid: "bc212218a4234f13a15d533ede7cced5" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38313 } } peers { permanent_uuid: "eb4cae636df24731b0b14518d325fec0" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33859 } }
I20260925 21:58:27.993180 15571 ts_tablet_manager.cc:1428] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:58:27.993227 15401 heartbeater.cc:503] Master 127.2.70.254:33101 was elected leader, sending a full tablet report...
I20260925 21:58:27.996583 15569 tablet_bootstrap.cc:492] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b: No bootstrap required, opened a new log
I20260925 21:58:27.996660 15569 ts_tablet_manager.cc:1397] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260925 21:58:27.999181 15569 raft_consensus.cc:348] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46795 } } peers { permanent_uuid: "bc212218a4234f13a15d533ede7cced5" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38313 } } peers { permanent_uuid: "eb4cae636df24731b0b14518d325fec0" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33859 } }
I20260925 21:58:27.999341 15569 raft_consensus.cc:374] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:27.999383 15569 raft_consensus.cc:729] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ac4bbb01684f6c8c8e85a264ebf16b, State: Initialized, Role: FOLLOWER
I20260925 21:58:27.999596 15569 consensus_queue.cc:260] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46795 } } peers { permanent_uuid: "bc212218a4234f13a15d533ede7cced5" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38313 } } peers { permanent_uuid: "eb4cae636df24731b0b14518d325fec0" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33859 } }
I20260925 21:58:27.999979 15569 ts_tablet_manager.cc:1428] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:58:28.000125 15535 heartbeater.cc:503] Master 127.2.70.254:33101 was elected leader, sending a full tablet report...
W20260925 21:58:28.042873 15268 tablet.cc:2369] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 21:58:28 dist-test-slave-2g0f krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373507, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for HTTP/127.2.70.212@KRBTEST.COM
I20260925 21:58:28.145120 15575 raft_consensus.cc:482] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:58:28.145287 15575 raft_consensus.cc:504] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46795 } } peers { permanent_uuid: "bc212218a4234f13a15d533ede7cced5" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38313 } } peers { permanent_uuid: "eb4cae636df24731b0b14518d325fec0" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33859 } }
I20260925 21:58:28.145751 15575 leader_election.cc:288] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4ac4bbb01684f6c8c8e85a264ebf16b (127.2.70.195:46795), bc212218a4234f13a15d533ede7cced5 (127.2.70.193:38313)
I20260925 21:58:28.149955 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:28.146124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:56285 (local address 127.2.70.195:46795)
0925 21:58:28.146168 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:28.146170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:28.146265 (+ 95us) server_negotiation.cc:407] Connection header received
0925 21:58:28.146369 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:28.146370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:28.146383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:28.146412 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:28.146646 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:28.147392 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:28.148929 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:28.149523 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:28.149577 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:28.149588 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:28.149830 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 21:58:28.149886 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 21:58:28.149971 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:28.145913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:46795 (local address 127.2.70.194:56285)
0925 21:58:28.146195 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:28.146207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:28.146293 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:28.146483 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:28.146484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:28.146485 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:28.146571 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:28.146573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:28.147491 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:28.147492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:28.148782 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:28.148787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:28.149667 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:28.149667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:28.149743 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:28.149747 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:58:28.149800 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 21:58:28.149842 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":81,"thread_start_us":88,"threads_started":1}
I20260925 21:58:28.150439 15489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab7529bc96a462d8d78a0dedcb12511" candidate_uuid: "eb4cae636df24731b0b14518d325fec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b" is_pre_election: true
I20260925 21:58:28.150638 15489 raft_consensus.cc:2454] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4cae636df24731b0b14518d325fec0 in term 0.
I20260925 21:58:28.150887 15291 leader_election.cc:302] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4ac4bbb01684f6c8c8e85a264ebf16b, eb4cae636df24731b0b14518d325fec0; no voters:
I20260925 21:58:28.151016 15575 raft_consensus.cc:2781] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:58:28.151053 15575 raft_consensus.cc:482] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:58:28.151074 15575 raft_consensus.cc:3037] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:28.152004 15575 raft_consensus.cc:504] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46795 } } peers { permanent_uuid: "bc212218a4234f13a15d533ede7cced5" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38313 } } peers { permanent_uuid: "eb4cae636df24731b0b14518d325fec0" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33859 } }
I20260925 21:58:28.152137 15575 leader_election.cc:288] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [CANDIDATE]: Term 1 election: Requested vote from peers c4ac4bbb01684f6c8c8e85a264ebf16b (127.2.70.195:46795), bc212218a4234f13a15d533ede7cced5 (127.2.70.193:38313)
I20260925 21:58:28.152571 15489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab7529bc96a462d8d78a0dedcb12511" candidate_uuid: "eb4cae636df24731b0b14518d325fec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b"
I20260925 21:58:28.152650 15489 raft_consensus.cc:3037] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:28.153544 15489 raft_consensus.cc:2454] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb4cae636df24731b0b14518d325fec0 in term 1.
I20260925 21:58:28.153786 15291 leader_election.cc:302] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4ac4bbb01684f6c8c8e85a264ebf16b, eb4cae636df24731b0b14518d325fec0; no voters:
I20260925 21:58:28.153934 15575 raft_consensus.cc:2781] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:58:28.154090 15575 raft_consensus.cc:686] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 1 LEADER]: Becoming Leader. State: Replica: eb4cae636df24731b0b14518d325fec0, State: Running, Role: LEADER
I20260925 21:58:28.154202 15575 consensus_queue.cc:237] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46795 } } peers { permanent_uuid: "bc212218a4234f13a15d533ede7cced5" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38313 } } peers { permanent_uuid: "eb4cae636df24731b0b14518d325fec0" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33859 } }
I20260925 21:58:28.155066 15069 catalog_manager.cc:5495] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb4cae636df24731b0b14518d325fec0 (127.2.70.194). New cstate: current_term: 1 leader_uuid: "eb4cae636df24731b0b14518d325fec0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc212218a4234f13a15d533ede7cced5" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb4cae636df24731b0b14518d325fec0" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33859 } health_report { overall_health: HEALTHY } } }
I20260925 21:58:28.160352 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:28.146162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:53303 (local address 127.2.70.193:38313)
0925 21:58:28.146214 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:28.146216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:28.146265 (+ 49us) server_negotiation.cc:407] Connection header received
0925 21:58:28.146381 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:28.146382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:28.146394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:28.146421 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:28.146646 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:28.147392 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:28.149228 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:28.149775 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:28.149828 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:28.149837 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:28.160202 (+ 10365us) server_negotiation.cc:299] Negotiation successful
0925 21:58:28.160274 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 21:58:28.160643 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:28.145988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:38313 (local address 127.2.70.194:53303)
0925 21:58:28.146210 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:28.146218 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:28.146293 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:28.146483 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:28.146484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:28.146487 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:28.146571 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:28.146574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:28.147491 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:28.147492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:28.148816 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:28.148821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:28.159990 (+ 11169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:28.159991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:28.160093 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:28.160099 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 21:58:28.160501 (+ 402us) client_negotiation.cc:240] Negotiation successful
0925 21:58:28.160560 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":69,"thread_start_us":70,"threads_started":1}
I20260925 21:58:28.161172 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab7529bc96a462d8d78a0dedcb12511" candidate_uuid: "eb4cae636df24731b0b14518d325fec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc212218a4234f13a15d533ede7cced5" is_pre_election: true
I20260925 21:58:28.161350 15220 raft_consensus.cc:2454] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4cae636df24731b0b14518d325fec0 in term 0.
I20260925 21:58:28.161501 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab7529bc96a462d8d78a0dedcb12511" candidate_uuid: "eb4cae636df24731b0b14518d325fec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc212218a4234f13a15d533ede7cced5"
I20260925 21:58:28.161559 15219 raft_consensus.cc:3037] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:28.162483 15219 raft_consensus.cc:2454] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb4cae636df24731b0b14518d325fec0 in term 1.
W20260925 21:58:28.178804 15536 tablet.cc:2369] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:58:28.203751 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:28.200284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.2.70.194:33859)
0925 21:58:28.200326 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:28.200328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:28.200337 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:58:28.200413 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:28.200414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:28.200439 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:28.200513 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:58:28.200699 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:28.201443 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:28.202675 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:28.202974 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:28.203037 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:28.203241 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:58:28.203390 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:58:28.203471 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:28.203645 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 21:58:28.203668 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 21:58:28.206187 15219 raft_consensus.cc:1261] T dab7529bc96a462d8d78a0dedcb12511 P bc212218a4234f13a15d533ede7cced5 [term 1 FOLLOWER]: Refusing update from remote peer eb4cae636df24731b0b14518d325fec0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:58:28.206321 15489 raft_consensus.cc:1261] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b [term 1 FOLLOWER]: Refusing update from remote peer eb4cae636df24731b0b14518d325fec0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:58:28.206527 15581 consensus_queue.cc:1035] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc212218a4234f13a15d533ede7cced5" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:58:28.206655 15575 consensus_queue.cc:1035] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4ac4bbb01684f6c8c8e85a264ebf16b" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:58:28.210160 15584 mvcc.cc:205] Tried to move back new op lower bound from 7333369889609379840 to 7333369889400184832. Current Snapshot: MvccSnapshot[applied={T|T < 7333369889609379840}]
I20260925 21:58:28.210772 15586 mvcc.cc:205] Tried to move back new op lower bound from 7333369889609379840 to 7333369889400184832. Current Snapshot: MvccSnapshot[applied={T|T < 7333369889609379840}]
I20260925 21:58:28.214334 15069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:58:28.214459 15069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:58:28.215204 15069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b4c5421e6f048e683329eaa9b4ba5d5: Sending DeleteTablet for 3 replicas of tablet dab7529bc96a462d8d78a0dedcb12511
I20260925 21:58:28.215548 15469 tablet_service.cc:1514] Processing DeleteTablet for tablet dab7529bc96a462d8d78a0dedcb12511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:28 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:40242
I20260925 21:58:28.216023 15335 tablet_service.cc:1514] Processing DeleteTablet for tablet dab7529bc96a462d8d78a0dedcb12511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:28 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:35554
I20260925 21:58:28.216028 15201 tablet_service.cc:1514] Processing DeleteTablet for tablet dab7529bc96a462d8d78a0dedcb12511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:28 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:39594
I20260925 21:58:28.216647 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 15107
I20260925 21:58:28.218964 15585 mvcc.cc:205] Tried to move back new op lower bound from 7333369889609379840 to 7333369889400184832. Current Snapshot: MvccSnapshot[applied={T|T < 7333369889609379840}]
I20260925 21:58:28.219092 15592 tablet_replica.cc:330] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b: stopping tablet replica
I20260925 21:58:28.219180 15592 raft_consensus.cc:2229] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:58:28.219244 15592 raft_consensus.cc:2258] T dab7529bc96a462d8d78a0dedcb12511 P c4ac4bbb01684f6c8c8e85a264ebf16b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:58:28.219997 15593 tablet_replica.cc:330] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0: stopping tablet replica
I20260925 21:58:28.220093 15593 raft_consensus.cc:2229] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:58:28.220208 15593 raft_consensus.cc:2258] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:58:28.220817 15593 ts_tablet_manager.cc:1905] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:58:28.222976 15593 ts_tablet_manager.cc:1918] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:58:28.223105 15593 log.cc:1196] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/dab7529bc96a462d8d78a0dedcb12511
I20260925 21:58:28.223462 15593 ts_tablet_manager.cc:1939] T dab7529bc96a462d8d78a0dedcb12511 P eb4cae636df24731b0b14518d325fec0: Deleting consensus metadata
I20260925 21:58:28.224280 15055 catalog_manager.cc:4841] TS eb4cae636df24731b0b14518d325fec0 (127.2.70.194:33859): tablet dab7529bc96a462d8d78a0dedcb12511 (table test-table [id=140a765878dc4446838a2122488414e4]) successfully deleted
W20260925 21:58:28.224439 15055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:38313 (error 108)
I20260925 21:58:28.224592 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 15269
I20260925 21:58:28.226521 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:28.225051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:38313 (local address 127.0.0.1:39598)
0925 21:58:28.226435 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:28.226482 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:38313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1359}
W20260925 21:58:28.226747 15055 catalog_manager.cc:4584] TS bc212218a4234f13a15d533ede7cced5 (127.2.70.193:38313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dab7529bc96a462d8d78a0dedcb12511: Network error: Client connection negotiation failed: client connection to 127.2.70.193:38313: connect: Connection refused (error 111)
I20260925 21:58:28.232669 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 15404
I20260925 21:58:28.239238 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:28.239113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:46795 (local address 127.0.0.1:40254)
0925 21:58:28.239158 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:28.239191 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.195:46795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 21:58:28.239506 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 15036
I20260925 21:58:28.247658 2331 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 21:58:28.582134 2331 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 21:58:28.582245 2331 mini_ranger.cc:66] Stopping Ranger...
I20260925 21:58:28.916585 2331 mini_ranger.cc:68] Stopped Ranger
2026-09-25 21:58:28.916 UTC [14103] LOG: received smart shutdown request
2026-09-25 21:58:28.920 UTC [14103] LOG: background worker "logical replication launcher" (PID 14112) exited with exit code 1
2026-09-25 21:58:28.921 UTC [14107] LOG: shutting down
2026-09-25 21:58:29.096 UTC [14103] LOG: database system is shut down
2026-09-25T21:58:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48202 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:58:29 dist-test-slave-2g0f krb5kdc[15605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:58:29 dist-test-slave-2g0f krb5kdc[15605](info): set up 2 sockets
Sep 25 21:58:29 dist-test-slave-2g0f krb5kdc[15605](info): commencing operation
krb5kdc: starting...
W20260925 21:58:30.135672 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:58:30.251665 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:33925
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:33925,127.2.70.253:45195,127.2.70.252:35427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:30.333164 15618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:30.333375 15618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:30.333407 15618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:30.334558 15618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:58:30.334599 15618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:30.334617 15618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:30.334633 15618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:58:30.334649 15618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:58:30.336196 15618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:33925,127.2.70.253:45195,127.2.70.252:35427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:33925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:30.336444 15618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:30.336647 15618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:30.341487 15626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:30.341498 15623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:30.341495 15624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:30.341743 15618 server_base.cc:1034] running on GCE node
I20260925 21:58:30.342038 15618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:30.342310 15618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:30.343454 15618 hybrid_clock.cc:648] HybridClock initialized: now 1790373510343437 us; error 27 us; skew 500 ppm
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:30.349176 15618 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:58:30.349671 15618 webserver.cc:460] Webserver started at http://127.2.70.254:41795/ using document root <none> and password file <none>
I20260925 21:58:30.349895 15618 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:30.349939 15618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:30.350057 15618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:58:30.351146 15618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "2ea890a7570847c0be2c4e6e26517785"
format_stamp: "Formatted at 2026-09-25 21:58:30 on dist-test-slave-2g0f"
server_key: "7afc54bb5a12570a326a8cf5ef352269"
server_key_iv: "e5132332f8ea3b72d0fe054922cf5098"
server_key_version: "encryptionkey@0"
I20260925 21:58:30.351486 15618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "2ea890a7570847c0be2c4e6e26517785"
format_stamp: "Formatted at 2026-09-25 21:58:30 on dist-test-slave-2g0f"
server_key: "7afc54bb5a12570a326a8cf5ef352269"
server_key_iv: "e5132332f8ea3b72d0fe054922cf5098"
server_key_version: "encryptionkey@0"
I20260925 21:58:30.352833 15618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:58:30.353587 15633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:30.353754 15618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:30.353826 15618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "2ea890a7570847c0be2c4e6e26517785"
format_stamp: "Formatted at 2026-09-25 21:58:30 on dist-test-slave-2g0f"
server_key: "7afc54bb5a12570a326a8cf5ef352269"
server_key_iv: "e5132332f8ea3b72d0fe054922cf5098"
server_key_version: "encryptionkey@0"
I20260925 21:58:30.353895 15618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:30.363050 15618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:30.364979 15618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:30.365104 15618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:30.369139 15618 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:33925
I20260925 21:58:30.369174 15685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:33925 every 8 connection(s)
I20260925 21:58:30.369506 15618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:58:30.370396 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:30.371946 15686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } has no permanent_uuid. Determining permanent_uuid...
I20260925 21:58:30.376700 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 15618
I20260925 21:58:30.376798 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:58:30.376973 2331 external_mini_cluster.cc:1428] Setting key 50d67e9170387d201840a6dfc51f0843
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:30.381755 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.372611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:33925 (local address 127.0.0.1:35892)
0925 21:58:30.372753 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.372787 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:30.373268 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:30.373549 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:30.373551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:30.373796 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:30.373973 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.373978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.374986 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.374987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.375688 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.375690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.376226 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.376227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.376285 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.376802 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:30.376810 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:30.378303 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:30.380226 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.380232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.380234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.380458 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.380703 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.380704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.380704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.380776 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.381209 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:30.381216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:30.381281 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 21:58:30.381384 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 21:58:30.381530 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260925 21:58:30.381767 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.372762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.2.70.254:33925)
0925 21:58:30.372871 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.372875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.372902 (+ 27us) server_negotiation.cc:407] Connection header received
0925 21:58:30.373431 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.373433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.373451 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.373534 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.374084 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.374898 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.375760 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.376155 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.376237 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.378390 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.378397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.378397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.378403 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.380151 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.380535 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.380538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.380538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.380644 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.380829 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.380830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.380831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.381117 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.381207 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.381446 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.381607 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":82,"thread_start_us":37,"threads_started":1}
I20260925 21:58:30.382527 15686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } has no permanent_uuid. Determining permanent_uuid...
I20260925 21:58:30.382876 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.382759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:45195 (local address 127.0.0.1:38078)
0925 21:58:30.382801 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.382850 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 21:58:30.383018 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.253:45195: connect: Connection refused (error 111)
I20260925 21:58:30.383404 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.383314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:45195 (local address 127.0.0.1:38092)
0925 21:58:30.383356 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.383382 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 21:58:30.383505 15686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.253:45195: Network error: Client connection negotiation failed: client connection to 127.2.70.253:45195: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.70.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.253.keytab.
Entry for principal kudu/127.2.70.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.253.keytab.
I20260925 21:58:30.403615 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.253:45195
--webserver_interface=127.2.70.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.253.keytab
--principal=kudu/127.2.70.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:33925,127.2.70.253:45195,127.2.70.252:35427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:58:30.432747 15686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } attempt: 1
I20260925 21:58:30.433275 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.433125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:45195 (local address 127.0.0.1:38104)
0925 21:58:30.433199 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.433241 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 21:58:30.433635 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.433549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:45195 (local address 127.0.0.1:38110)
0925 21:58:30.433591 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.433615 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 21:58:30.433725 15686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.253:45195: Network error: Client connection negotiation failed: client connection to 127.2.70.253:45195: connect: Connection refused (error 111)
W20260925 21:58:30.484467 15693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:30.484683 15693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:30.484717 15693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:30.485847 15693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:58:30.485890 15693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:30.485913 15693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:30.485932 15693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:58:30.485952 15693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:58:30.487430 15693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:33925,127.2.70.253:45195,127.2.70.252:35427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.253.keytab
--principal=kudu/127.2.70.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.253:45195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.70.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:30.487671 15693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:30.487939 15693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:58:30.492683 15693 server_base.cc:1034] running on GCE node
W20260925 21:58:30.492664 15701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:30.492630 15699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:30.492643 15698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:30.493185 15693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:30.493455 15693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:30.494629 15693 hybrid_clock.cc:648] HybridClock initialized: now 1790373510494619 us; error 28 us; skew 500 ppm
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:30.497145 15693 init.cc:377] Logged in from keytab as kudu/127.2.70.253@KRBTEST.COM (short username kudu)
I20260925 21:58:30.497553 15693 webserver.cc:460] Webserver started at http://127.2.70.253:37795/ using document root <none> and password file <none>
I20260925 21:58:30.497753 15693 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:30.497799 15693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:30.497942 15693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:58:30.498993 15693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/data/instance:
uuid: "4d39bbe2ef114a6baaa76587f7891d64"
format_stamp: "Formatted at 2026-09-25 21:58:30 on dist-test-slave-2g0f"
server_key: "4e65204dd4062902ad2f06f847bbba0b"
server_key_iv: "615d138d62c94b1b3b06f004b012fd9f"
server_key_version: "encryptionkey@0"
I20260925 21:58:30.499315 15693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/wal/instance:
uuid: "4d39bbe2ef114a6baaa76587f7891d64"
format_stamp: "Formatted at 2026-09-25 21:58:30 on dist-test-slave-2g0f"
server_key: "4e65204dd4062902ad2f06f847bbba0b"
server_key_iv: "615d138d62c94b1b3b06f004b012fd9f"
server_key_version: "encryptionkey@0"
I20260925 21:58:30.500586 15693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 21:58:30.501317 15708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:30.501480 15693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:30.501567 15693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/wal
uuid: "4d39bbe2ef114a6baaa76587f7891d64"
format_stamp: "Formatted at 2026-09-25 21:58:30 on dist-test-slave-2g0f"
server_key: "4e65204dd4062902ad2f06f847bbba0b"
server_key_iv: "615d138d62c94b1b3b06f004b012fd9f"
server_key_version: "encryptionkey@0"
I20260925 21:58:30.501627 15693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:30.501853 15686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } attempt: 2
I20260925 21:58:30.502274 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.502139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:45195 (local address 127.0.0.1:38124)
0925 21:58:30.502197 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.502237 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 21:58:30.502563 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.502486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:45195 (local address 127.0.0.1:38132)
0925 21:58:30.502519 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.502543 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 21:58:30.502645 15686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.253:45195: Network error: Client connection negotiation failed: client connection to 127.2.70.253:45195: connect: Connection refused (error 111)
I20260925 21:58:30.512830 15693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:30.514776 15693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:30.514899 15693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:30.518548 15693 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.253:45195
I20260925 21:58:30.518584 15760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.253:45195 every 8 connection(s)
I20260925 21:58:30.518874 15693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
I20260925 21:58:30.519718 15761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:30.521265 15761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.253@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:30.528388 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 15693
I20260925 21:58:30.528489 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-1/wal/instance
I20260925 21:58:30.528674 2331 external_mini_cluster.cc:1428] Setting key 644f0a67fe2c032887052cd26d919021
I20260925 21:58:30.529789 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.521924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.2.70.254:33925)
0925 21:58:30.521975 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.521977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.522166 (+ 189us) server_negotiation.cc:407] Connection header received
0925 21:58:30.522595 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.522596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.522607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.522679 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.523296 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.523803 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.524835 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.525051 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.525115 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.527062 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.527063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.527063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.527070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.528542 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.528932 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.528933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.528934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.528998 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.529206 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.529207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.529207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.529333 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.529401 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.529612 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.529692 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 21:58:30.529902 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.521879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:33925 (local address 127.0.0.1:35906)
0925 21:58:30.522029 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.522069 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:30.522510 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:30.522768 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:30.522772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:30.523020 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:30.523213 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.523218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.523945 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.523946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.524744 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.524746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.525125 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.525125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.525227 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.525666 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:30.525673 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:30.526964 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:30.528624 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.528630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.528632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.528866 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.529059 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.529059 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.529060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.529157 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.529390 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:30.529393 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:30.529510 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 21:58:30.529587 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 21:58:30.529722 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260925 21:58:30.530449 15761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.253@KRBTEST.COM for kudu/127.2.70.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.252@KRBTEST.COM; defaulting to no policy
I20260925 21:58:30.537037 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.530714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:45195 (local address 127.0.0.1:38142)
0925 21:58:30.530757 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.530768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:30.530810 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:30.531032 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:30.531033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:30.531168 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:30.531211 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.531212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.532049 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.532050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.532538 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.532539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.532974 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.532975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.533037 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.533378 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:30.533383 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:30.534539 (+ 1156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:30.536103 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.536104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.536104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.536249 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.536426 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.536427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.536428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.536494 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.536739 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:30.536739 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:30.536777 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 21:58:30.536832 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 21:58:30.536944 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 21:58:30.537122 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.530750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.2.70.253:45195)
0925 21:58:30.530888 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.530892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.530905 (+ 13us) server_negotiation.cc:407] Connection header received
0925 21:58:30.530923 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.530924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.530936 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.531004 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.531271 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.531981 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.532601 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.532884 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.532941 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.534617 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.534624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.534624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.534630 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.536038 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.536310 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.536312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.536313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.536354 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.536555 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.536555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.536556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.536681 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.536754 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.536855 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.536989 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260925 21:58:30.537674 15761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } has no permanent_uuid. Determining permanent_uuid...
I20260925 21:58:30.538077 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.537884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:46564)
0925 21:58:30.537984 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.538050 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.252@KRBTEST.COM" created.
W20260925 21:58:30.538237 15709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
I20260925 21:58:30.538599 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.538519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:46576)
0925 21:58:30.538553 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.538577 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 21:58:30.538688 15761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:35427: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.252.keytab.
Entry for principal kudu/127.2.70.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.252.keytab.
I20260925 21:58:30.552978 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.252:35427
--webserver_interface=127.2.70.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.252.keytab
--principal=kudu/127.2.70.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:33925,127.2.70.253:45195,127.2.70.252:35427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:58:30.587950 15761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } attempt: 1
I20260925 21:58:30.588505 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.588337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:46590)
0925 21:58:30.588416 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.588467 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 21:58:30.588805 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.588724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:46594)
0925 21:58:30.588760 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.588785 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 21:58:30.588943 15761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:35427: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
I20260925 21:58:30.593786 15686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } attempt: 3
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for kudu/127.2.70.253@KRBTEST.COM
I20260925 21:58:30.600610 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.594081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:45195 (local address 127.0.0.1:38154)
0925 21:58:30.594130 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.594142 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:30.594186 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:30.594319 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:30.594320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:30.594495 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:30.594562 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.594564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.595305 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.595306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.595822 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.595824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.596192 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.596193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.596266 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.596729 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:30.596733 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:30.597941 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:30.599571 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.599572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.599573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.599787 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.600034 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.600035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.600036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.600099 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.600301 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:30.600302 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:30.600349 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 21:58:30.600421 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 21:58:30.600513 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 21:58:30.600608 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.594148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.2.70.253:45195)
0925 21:58:30.594203 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.594205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.594214 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:58:30.594249 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.594254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.594265 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.594314 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.594644 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.595217 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.595922 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.596134 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.596190 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.598023 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.598024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.598025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.598031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.599480 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.599916 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.599917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.599917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.599974 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.600150 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.600150 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.600150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.600249 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.600305 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.600472 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.600544 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 21:58:30.601063 15686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } has no permanent_uuid. Determining permanent_uuid...
I20260925 21:58:30.601367 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.601275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:46608)
0925 21:58:30.601315 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.601345 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 21:58:30.601732 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.601655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:46614)
0925 21:58:30.601692 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.601715 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 21:58:30.601817 15686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:35427: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
I20260925 21:58:30.633006 15686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } attempt: 1
I20260925 21:58:30.633543 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.633365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:46618)
0925 21:58:30.633466 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.633510 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 21:58:30.633986 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.633818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:46632)
0925 21:58:30.633883 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.633913 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 21:58:30.634083 15686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:35427: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
W20260925 21:58:30.634208 15768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:30.634404 15768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:30.634436 15768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:30.635601 15768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:58:30.635641 15768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:30.635659 15768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:30.635676 15768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:58:30.635700 15768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:58:30.637241 15768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:33925,127.2.70.253:45195,127.2.70.252:35427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.252.keytab
--principal=kudu/127.2.70.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.252:35427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.70.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:30.637491 15768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:30.637696 15768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:30.642386 15776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:30.642416 15773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:30.642385 15774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:30.642457 15768 server_base.cc:1034] running on GCE node
I20260925 21:58:30.642870 15768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:30.643189 15768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:30.644346 15768 hybrid_clock.cc:648] HybridClock initialized: now 1790373510644327 us; error 33 us; skew 500 ppm
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:30.646831 15768 init.cc:377] Logged in from keytab as kudu/127.2.70.252@KRBTEST.COM (short username kudu)
I20260925 21:58:30.647301 15768 webserver.cc:460] Webserver started at http://127.2.70.252:42323/ using document root <none> and password file <none>
I20260925 21:58:30.647513 15768 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:30.647557 15768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:30.647675 15768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:58:30.648761 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/data/instance:
uuid: "b98659f5e6f348b79c7baaee6eacfd68"
format_stamp: "Formatted at 2026-09-25 21:58:30 on dist-test-slave-2g0f"
server_key: "cf781254f7371d0e5ef354ca16a0ae00"
server_key_iv: "aa5804a6a0209cbb1ec9709b9d7cf524"
server_key_version: "encryptionkey@0"
I20260925 21:58:30.649081 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/wal/instance:
uuid: "b98659f5e6f348b79c7baaee6eacfd68"
format_stamp: "Formatted at 2026-09-25 21:58:30 on dist-test-slave-2g0f"
server_key: "cf781254f7371d0e5ef354ca16a0ae00"
server_key_iv: "aa5804a6a0209cbb1ec9709b9d7cf524"
server_key_version: "encryptionkey@0"
I20260925 21:58:30.650302 15768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:58:30.651006 15783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:30.651178 15768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:58:30.651275 15768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/wal
uuid: "b98659f5e6f348b79c7baaee6eacfd68"
format_stamp: "Formatted at 2026-09-25 21:58:30 on dist-test-slave-2g0f"
server_key: "cf781254f7371d0e5ef354ca16a0ae00"
server_key_iv: "aa5804a6a0209cbb1ec9709b9d7cf524"
server_key_version: "encryptionkey@0"
I20260925 21:58:30.651351 15768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:30.657104 15761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } attempt: 2
I20260925 21:58:30.657521 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.657403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:44806)
0925 21:58:30.657457 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.657490 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 21:58:30.657910 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.657752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:44822)
0925 21:58:30.657822 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.657877 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 21:58:30.658051 15761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:35427: Network error: Client connection negotiation failed: client connection to 127.2.70.252:35427: connect: Connection refused (error 111)
I20260925 21:58:30.678210 15768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:30.680377 15768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:30.680503 15768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:30.684296 15768 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.252:35427
I20260925 21:58:30.684337 15835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.252:35427 every 8 connection(s)
I20260925 21:58:30.684669 15768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
I20260925 21:58:30.685555 15836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:30.687071 15836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } has no permanent_uuid. Determining permanent_uuid...
I20260925 21:58:30.687749 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 15768
I20260925 21:58:30.687865 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790373357410105-2331-0/minicluster-data/master-2/wal/instance
I20260925 21:58:30.688052 2331 external_mini_cluster.cc:1428] Setting key e552387edd1d372474d97ee03c8a842a
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.252@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.252@KRBTEST.COM
I20260925 21:58:30.696182 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.687861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.2.70.254:33925)
0925 21:58:30.687911 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.687914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.688117 (+ 203us) server_negotiation.cc:407] Connection header received
0925 21:58:30.688694 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.688695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.688707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.688755 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.689289 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.689765 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.690661 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.690862 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.690926 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.693249 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.693250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.693251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.693259 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.694776 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.695187 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.695189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.695189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.695242 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.695471 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.695472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.695473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.695589 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.695643 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.696013 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.696085 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 21:58:30.696306 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.687769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:33925 (local address 127.0.0.1:36230)
0925 21:58:30.687985 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.688026 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:30.688588 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:30.688773 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:30.688776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:30.689043 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:30.689196 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.689200 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.689832 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.689833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.690564 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.690566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.690928 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.690929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.691058 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.691530 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:30.691541 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:30.693158 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:30.694863 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.694870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.694872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.695106 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.695306 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.695307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.695307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.695412 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.695644 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:30.695650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:30.695793 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 21:58:30.695919 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 21:58:30.696084 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":22,"thread_start_us":95,"threads_started":1}
I20260925 21:58:30.696609 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.688436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.2.70.252:35427)
0925 21:58:30.688574 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.688579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.688613 (+ 34us) server_negotiation.cc:407] Connection header received
0925 21:58:30.688692 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.688693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.688709 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.688781 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.689401 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.690261 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.690965 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.691265 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.691323 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.693690 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.693698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.693698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.693706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.695260 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.695635 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.695639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.695639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.695699 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.695931 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.695932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.695933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.696080 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.696237 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.696310 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.696390 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260925 21:58:30.696955 15836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.252@KRBTEST.COM for kudu/127.2.70.253@KRBTEST.COM
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:30.703568 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.697215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:45195 (local address 127.0.0.1:53758)
0925 21:58:30.697437 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.697445 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:30.697480 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:30.697591 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:30.697592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:30.697706 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:30.697757 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.697759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.698433 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.698434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.698925 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.698926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.699274 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.699275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.699346 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.699736 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:30.699741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:30.700992 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:30.702521 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.702522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.702522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.702685 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.702959 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.702959 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.702960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.703023 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.703279 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:30.703279 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:30.703321 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 21:58:30.703380 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 21:58:30.703467 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 21:58:30.703581 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.697288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.2.70.253:45195)
0925 21:58:30.697347 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.697349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.697474 (+ 125us) server_negotiation.cc:407] Connection header received
0925 21:58:30.697524 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.697524 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.697534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.697587 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.697853 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.698322 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.699002 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.699204 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.699260 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.701087 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.701089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.701089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.701096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.702442 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.702763 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.702764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.702764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.702823 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.703096 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.703097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.703097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.703219 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.703282 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.703423 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.703495 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 21:58:30.703974 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.697386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.2.70.254:33925)
0925 21:58:30.697431 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.697433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.697439 (+ 6us) server_negotiation.cc:407] Connection header received
0925 21:58:30.697457 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.697458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.697467 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.697523 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.698055 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.698504 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.699141 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.699333 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.699384 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.701697 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.701698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.701698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.701705 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.702979 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.703245 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.703246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.703246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.703295 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.703472 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.703473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.703473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.703577 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.703746 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.703780 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.703873 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 21:58:30.704378 15836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.252@KRBTEST.COM for kudu/127.2.70.252@KRBTEST.COM
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.253@KRBTEST.COM
I20260925 21:58:30.709201 15686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } attempt: 2
I20260925 21:58:30.711535 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.704636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:44832)
0925 21:58:30.704933 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.704944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:30.704997 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:30.705126 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:30.705127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:30.705262 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:30.705299 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.705300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.706032 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.706033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.706745 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.706747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.707194 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.707205 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.707253 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.707632 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:30.707636 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:30.708888 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:30.710495 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.710496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.710497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.710693 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.710899 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.710900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.710900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.710960 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.711205 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:30.711206 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:30.711249 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 21:58:30.711316 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 21:58:30.711412 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260925 21:58:30.711715 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.704665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.2.70.252:35427)
0925 21:58:30.704695 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.704697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.705013 (+ 316us) server_negotiation.cc:407] Connection header received
0925 21:58:30.705057 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.705058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.705070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.705124 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.705386 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.705859 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.706815 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.707124 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.707177 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.708963 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.708964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.708964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.708970 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.710430 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.710766 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.710767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.710767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.710833 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.711027 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.711027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.711028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.711145 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.711206 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.711342 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.711470 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":14}
I20260925 21:58:30.712638 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.704662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.2.70.253:45195)
0925 21:58:30.704705 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.704706 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.704713 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:58:30.704744 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.704745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.704754 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.704801 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.705395 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.705839 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.706742 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.707074 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.707142 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.709787 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.709788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.709789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.709795 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.711175 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.711536 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.711536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.711537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.711588 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.711870 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.711870 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.711871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.711992 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.712434 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.712474 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.712546 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 21:58:30.713589 15836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68: Bootstrap starting.
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for kudu/127.2.70.252@KRBTEST.COM
I20260925 21:58:30.714296 15836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:30.714509 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 21:58:30.714654 15836 log.cc:824] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:30.715449 15836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68: No bootstrap required, opened a new log
I20260925 21:58:30.717078 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.709454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:44834)
0925 21:58:30.709503 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.709513 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:30.709548 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:30.710158 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:30.710159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:30.710275 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:30.710319 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.710321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.711587 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.711588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.712309 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.712312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.712650 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.712651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.712705 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.713041 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:30.713046 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:30.714249 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:30.715996 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.715997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.715998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.716200 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.716448 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.716449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.716449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.716510 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.716779 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:30.716780 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:30.716824 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 21:58:30.716886 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 21:58:30.716981 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 21:58:30.717106 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.709489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.2.70.252:35427)
0925 21:58:30.709877 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.709879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.709885 (+ 6us) server_negotiation.cc:407] Connection header received
0925 21:58:30.709903 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.709904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.709912 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.709970 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.710400 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.711526 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.712386 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.712585 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.712637 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.714326 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.714327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.714327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.714335 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.715903 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.716345 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.716346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.716346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.716401 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.716574 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.716575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.716575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.716718 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.716774 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.716914 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.716989 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":332,"threads_started":1}
I20260925 21:58:30.717617 15836 raft_consensus.cc:348] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } }
I20260925 21:58:30.717774 15836 raft_consensus.cc:374] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:30.717839 15836 raft_consensus.cc:729] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98659f5e6f348b79c7baaee6eacfd68, State: Initialized, Role: FOLLOWER
I20260925 21:58:30.717980 15836 consensus_queue.cc:260] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } }
I20260925 21:58:30.718303 15836 sys_catalog.cc:564] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:58:30.719004 15686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785: Bootstrap starting.
I20260925 21:58:30.720319 15686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785: Neither blocks nor log segments found. Creating new log.
W20260925 21:58:30.720342 15861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:58:30.720386 15861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:58:30.720667 15686 log.cc:824] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:30.721017 15850 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } } }
I20260925 21:58:30.721230 15850 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 21:58:30.720573 15768 master_runner.cc:186] Error getting master registration for 127.2.70.254:33925: OK, instance_id { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" instance_seqno: 1790373510367969 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 21:58:30.723311 15768 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 21:58:30.725289 15686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785: No bootstrap required, opened a new log
I20260925 21:58:30.727025 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.716144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.2.70.254:33925)
0925 21:58:30.716181 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.716183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.716487 (+ 304us) server_negotiation.cc:407] Connection header received
0925 21:58:30.716534 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.716535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.716546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.716594 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.717329 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.717869 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.718668 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.718916 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.718975 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.723660 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.723661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.723662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.723669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.725653 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.726033 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.726034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.726035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.726090 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.726295 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.726295 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.726296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.726423 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.726533 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.726862 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.726935 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 21:58:30.727090 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.716264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.2.70.252:35427)
0925 21:58:30.716300 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.716302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.719784 (+ 3482us) server_negotiation.cc:407] Connection header received
0925 21:58:30.719912 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.719913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.719923 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.719952 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.720872 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.721471 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.722145 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.722417 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.722474 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.724206 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.724207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.724207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.724214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.725824 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.726191 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.726192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.726192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.726248 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.726452 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.726453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.726453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.726575 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.726636 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.726919 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.727004 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 21:58:30.727414 15686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } }
I20260925 21:58:30.727552 15686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:30.727588 15686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea890a7570847c0be2c4e6e26517785, State: Initialized, Role: FOLLOWER
I20260925 21:58:30.727711 15686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } }
I20260925 21:58:30.728106 15686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 21:58:30.730149 15875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:58:30.730234 15875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:58:30.730535 15864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } } }
I20260925 21:58:30.732193 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.716344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.2.70.253:45195)
0925 21:58:30.716376 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.716378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.722226 (+ 5848us) server_negotiation.cc:407] Connection header received
0925 21:58:30.722311 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.722312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.722323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.722351 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.722973 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.723666 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.724617 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.724910 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.724980 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.729432 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.729433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.729434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.729442 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.731016 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.731392 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.731393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.731394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.731453 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.731632 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.731633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.731633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.731759 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.731895 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.732007 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.732092 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 21:58:30.730834 15618 master_runner.cc:186] Error getting master registration for 127.2.70.253:45195: OK, instance_id { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" instance_seqno: 1790373510517535 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 21:58:30.733295 15618 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 21:58:30.733224 15864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 21:58:30.749214 15761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } attempt: 3
Sep 25 21:58:30 dist-test-slave-2g0f krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.253@KRBTEST.COM for kudu/127.2.70.252@KRBTEST.COM
I20260925 21:58:30.757443 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.749533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.2.70.252:35427)
0925 21:58:30.749572 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:30.749574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:30.749582 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:58:30.749645 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:30.749645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:30.749655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:30.749705 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:30.750056 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.750821 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.751798 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.752125 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.752197 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.754406 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:30.754408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:30.754408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:30.754416 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:30.756216 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.756619 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.756620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.756620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.756679 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:30.756919 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:30.756920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:30.756920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:30.757059 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:30.757125 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:30.757270 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 21:58:30.757344 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 21:58:30.757455 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:30.749489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:35427 (local address 127.0.0.1:44850)
0925 21:58:30.749536 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:30.749548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:30.749583 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:30.749714 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:30.749715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:30.749925 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:30.749978 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.749981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.750906 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.750908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.751717 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:30.751719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:30.752196 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:30.752196 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:30.752258 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:30.752742 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:30.752747 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:30.754312 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:30.756288 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.756289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.756290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.756547 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.756786 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:30.756787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:30.756788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:30.756857 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:30.757123 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:30.757123 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:30.757173 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 21:58:30.757242 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 21:58:30.757344 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 21:58:30.759554 15761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64: Bootstrap starting.
I20260925 21:58:30.760334 15761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:30.760635 15761 log.cc:824] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:30.761531 15761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64: No bootstrap required, opened a new log
I20260925 21:58:30.763614 15761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } }
I20260925 21:58:30.763752 15761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:30.763794 15761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d39bbe2ef114a6baaa76587f7891d64, State: Initialized, Role: FOLLOWER
I20260925 21:58:30.763996 15761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } }
I20260925 21:58:30.764631 15850 raft_consensus.cc:482] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:58:30.764729 15850 raft_consensus.cc:504] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } }
I20260925 21:58:30.764981 15850 leader_election.cc:288] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ea890a7570847c0be2c4e6e26517785 (127.2.70.254:33925), 4d39bbe2ef114a6baaa76587f7891d64 (127.2.70.253:45195)
I20260925 21:58:30.765589 15660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b98659f5e6f348b79c7baaee6eacfd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ea890a7570847c0be2c4e6e26517785" is_pre_election: true
I20260925 21:58:30.765743 15660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98659f5e6f348b79c7baaee6eacfd68 in term 0.
I20260925 21:58:30.766685 15785 leader_election.cc:302] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea890a7570847c0be2c4e6e26517785, b98659f5e6f348b79c7baaee6eacfd68; no voters:
I20260925 21:58:30.766968 15850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:58:30.767031 15850 raft_consensus.cc:482] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:58:30.767053 15850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:30.768282 15850 raft_consensus.cc:504] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } }
I20260925 21:58:30.768445 15850 leader_election.cc:288] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [CANDIDATE]: Term 1 election: Requested vote from peers 2ea890a7570847c0be2c4e6e26517785 (127.2.70.254:33925), 4d39bbe2ef114a6baaa76587f7891d64 (127.2.70.253:45195)
I20260925 21:58:30.768924 15660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b98659f5e6f348b79c7baaee6eacfd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ea890a7570847c0be2c4e6e26517785"
I20260925 21:58:30.769002 15660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:30.769831 15660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98659f5e6f348b79c7baaee6eacfd68 in term 1.
I20260925 21:58:30.770051 15785 leader_election.cc:302] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea890a7570847c0be2c4e6e26517785, b98659f5e6f348b79c7baaee6eacfd68; no voters:
I20260925 21:58:30.770140 15850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:58:30.770299 15850 raft_consensus.cc:686] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [term 1 LEADER]: Becoming Leader. State: Replica: b98659f5e6f348b79c7baaee6eacfd68, State: Running, Role: LEADER
I20260925 21:58:30.770417 15850 consensus_queue.cc:237] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } }
I20260925 21:58:30.770884 15879 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b98659f5e6f348b79c7baaee6eacfd68. Latest consensus state: current_term: 1 leader_uuid: "b98659f5e6f348b79c7baaee6eacfd68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } } }
I20260925 21:58:30.771025 15761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:58:30.771106 15735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b98659f5e6f348b79c7baaee6eacfd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d39bbe2ef114a6baaa76587f7891d64" is_pre_election: true
I20260925 21:58:30.771234 15735 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98659f5e6f348b79c7baaee6eacfd68 in term 0.
I20260925 21:58:30.771325 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b98659f5e6f348b79c7baaee6eacfd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d39bbe2ef114a6baaa76587f7891d64"
I20260925 21:58:30.771370 15734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:30.772198 15734 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98659f5e6f348b79c7baaee6eacfd68 in term 1.
I20260925 21:58:30.772562 15879 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:30.772804 15892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:58:30.773011 15892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:58:30.773078 15878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } } }
I20260925 21:58:30.774020 15878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 21:58:30.776568 15734 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [term 1 FOLLOWER]: Refusing update from remote peer b98659f5e6f348b79c7baaee6eacfd68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:58:30.776675 15660 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [term 1 FOLLOWER]: Refusing update from remote peer b98659f5e6f348b79c7baaee6eacfd68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:58:30.776831 15879 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:58:30.777046 15879 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:58:30.777937 15878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b98659f5e6f348b79c7baaee6eacfd68. Latest consensus state: current_term: 1 leader_uuid: "b98659f5e6f348b79c7baaee6eacfd68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } } }
I20260925 21:58:30.778014 15878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 21:58:30.778262 15864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b98659f5e6f348b79c7baaee6eacfd68. Latest consensus state: current_term: 1 leader_uuid: "b98659f5e6f348b79c7baaee6eacfd68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } } }
I20260925 21:58:30.778338 15864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 21:58:30.778839 15850 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b98659f5e6f348b79c7baaee6eacfd68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } } }
I20260925 21:58:30.778919 15850 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:30.779305 15892 catalog_manager.cc:1347] Generated new cluster ID: de8493dbd2f04999a1e33d66d8ca89a8
I20260925 21:58:30.779346 15892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:58:30.779376 15878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b98659f5e6f348b79c7baaee6eacfd68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } } }
I20260925 21:58:30.779443 15878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 21:58:30.780117 15894 mvcc.cc:205] Tried to move back new op lower bound from 7333369900138991616 to 7333369900116316160. Current Snapshot: MvccSnapshot[applied={T|T < 7333369900138991616}]
I20260925 21:58:30.780210 15893 mvcc.cc:205] Tried to move back new op lower bound from 7333369900138991616 to 7333369900116316160. Current Snapshot: MvccSnapshot[applied={T|T < 7333369900138991616 or (T in {7333369900138991616})}]
I20260925 21:58:30.784617 15879 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b98659f5e6f348b79c7baaee6eacfd68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } } }
I20260925 21:58:30.784706 15879 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:30.785030 15864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b98659f5e6f348b79c7baaee6eacfd68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea890a7570847c0be2c4e6e26517785" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33925 } } peers { permanent_uuid: "4d39bbe2ef114a6baaa76587f7891d64" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 45195 } } peers { permanent_uuid: "b98659f5e6f348b79c7baaee6eacfd68" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 35427 } } }
I20260925 21:58:30.785113 15864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 21:58:30.785781 15891 catalog_manager.cc:1259] Loaded cluster ID: de8493dbd2f04999a1e33d66d8ca89a8
I20260925 21:58:30.785831 15891 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64: loading cluster ID for follower catalog manager: success
W20260925 21:58:30.785928 15891 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260925 21:58:30.785965 15891 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:58:30.796269 15896 mvcc.cc:205] Tried to move back new op lower bound from 7333369900138991616 to 7333369900116316160. Current Snapshot: MvccSnapshot[applied={T|T < 7333369900138991616}]
I20260925 21:58:30.800261 15892 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:58:30.801020 15892 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:58:30.814567 15892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b98659f5e6f348b79c7baaee6eacfd68: Generated new TSK 0
I20260925 21:58:30.814837 15892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:58:31.730681 15875 catalog_manager.cc:1259] Loaded cluster ID: de8493dbd2f04999a1e33d66d8ca89a8
I20260925 21:58:31.730755 15875 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785: loading cluster ID for follower catalog manager: success
I20260925 21:58:31.731582 15875 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785: acquiring CA information for follower catalog manager: success
I20260925 21:58:31.731901 15875 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ea890a7570847c0be2c4e6e26517785: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 21:58:31.787070 15891 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64: acquiring CA information for follower catalog manager: success
I20260925 21:58:31.787380 15891 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d39bbe2ef114a6baaa76587f7891d64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 21:58:31.870666 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 15618
I20260925 21:58:31.875278 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 15693
I20260925 21:58:31.879401 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 15768
2026-09-25T21:58:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2698 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:58:31 dist-test-slave-2g0f krb5kdc[15909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:58:31 dist-test-slave-2g0f krb5kdc[15909](info): set up 2 sockets
Sep 25 21:58:31 dist-test-slave-2g0f krb5kdc[15909](info): commencing operation
krb5kdc: starting...
W20260925 21:58:32.831686 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:58:32 dist-test-slave-2g0f krb5kdc[15909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:58:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:58:32.948822 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:33299
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:33299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:33.028503 15922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:33.028702 15922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:33.028721 15922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:33.029875 15922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:58:33.029918 15922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:33.029932 15922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:33.029951 15922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:58:33.029963 15922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:58:33.031450 15922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:33299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:33299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:33.031667 15922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:33.031903 15922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:33.036463 15927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:33.036512 15928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:33.036474 15930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:33.036756 15922 server_base.cc:1034] running on GCE node
I20260925 21:58:33.036975 15922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:33.037235 15922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:33.038390 15922 hybrid_clock.cc:648] HybridClock initialized: now 1790373513038369 us; error 30 us; skew 500 ppm
Sep 25 21:58:33 dist-test-slave-2g0f krb5kdc[15909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373513, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:33.040702 15922 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:58:33.041127 15922 webserver.cc:460] Webserver started at http://127.2.70.254:43095/ using document root <none> and password file <none>
I20260925 21:58:33.041343 15922 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:33.041389 15922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:33.041507 15922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:58:33.042493 15922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "4758f38482394ede98f4fc463a7287cc"
format_stamp: "Formatted at 2026-09-25 21:58:33 on dist-test-slave-2g0f"
server_key: "9daa42ec0688442fb7f5af761705fb39"
server_key_iv: "b2d9d8b37520a8429c84cacefd488b1b"
server_key_version: "encryptionkey@0"
I20260925 21:58:33.042827 15922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "4758f38482394ede98f4fc463a7287cc"
format_stamp: "Formatted at 2026-09-25 21:58:33 on dist-test-slave-2g0f"
server_key: "9daa42ec0688442fb7f5af761705fb39"
server_key_iv: "b2d9d8b37520a8429c84cacefd488b1b"
server_key_version: "encryptionkey@0"
I20260925 21:58:33.044232 15922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 21:58:33.045007 15937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:33.045164 15922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:58:33.045248 15922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "4758f38482394ede98f4fc463a7287cc"
format_stamp: "Formatted at 2026-09-25 21:58:33 on dist-test-slave-2g0f"
server_key: "9daa42ec0688442fb7f5af761705fb39"
server_key_iv: "b2d9d8b37520a8429c84cacefd488b1b"
server_key_version: "encryptionkey@0"
I20260925 21:58:33.045313 15922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:33.057474 15922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:33.059499 15922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:33.059628 15922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:33.063395 15922 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:33299
I20260925 21:58:33.063457 15989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:33299 every 8 connection(s)
I20260925 21:58:33.063767 15922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:58:33.064636 15990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:33.066861 15990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc: Bootstrap starting.
I20260925 21:58:33.067550 15990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:33.067807 15990 log.cc:824] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:33.068606 15990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc: No bootstrap required, opened a new log
I20260925 21:58:33.070231 15990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4758f38482394ede98f4fc463a7287cc" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33299 } }
I20260925 21:58:33.070370 15990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:33.070391 15990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4758f38482394ede98f4fc463a7287cc, State: Initialized, Role: FOLLOWER
I20260925 21:58:33.070523 15990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4758f38482394ede98f4fc463a7287cc" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33299 } }
I20260925 21:58:33.070612 15990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:58:33.070636 15990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:58:33.070669 15990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:33.071193 15990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4758f38482394ede98f4fc463a7287cc" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33299 } }
I20260925 21:58:33.071283 15990 leader_election.cc:302] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4758f38482394ede98f4fc463a7287cc; no voters:
I20260925 21:58:33.071432 15990 leader_election.cc:288] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:58:33.071481 15993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:58:33.071623 15993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [term 1 LEADER]: Becoming Leader. State: Replica: 4758f38482394ede98f4fc463a7287cc, State: Running, Role: LEADER
I20260925 21:58:33.071679 15990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:58:33.071713 15993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4758f38482394ede98f4fc463a7287cc" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33299 } }
I20260925 21:58:33.072278 15994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4758f38482394ede98f4fc463a7287cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4758f38482394ede98f4fc463a7287cc" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33299 } } }
I20260925 21:58:33.072371 15994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:33.072319 15995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4758f38482394ede98f4fc463a7287cc. Latest consensus state: current_term: 1 leader_uuid: "4758f38482394ede98f4fc463a7287cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4758f38482394ede98f4fc463a7287cc" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 33299 } } }
I20260925 21:58:33.072721 16002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:58:33.072775 15995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:33.073257 16002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:58:33.073563 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 15922
I20260925 21:58:33.073647 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:58:33.073838 2331 external_mini_cluster.cc:1428] Setting key b78068c62ca26e059ddf855c3d2fd113
I20260925 21:58:33.075450 16002 catalog_manager.cc:1347] Generated new cluster ID: 78fd074859454d0888849f5621887e01
I20260925 21:58:33.075491 16002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:58:33.080266 16002 catalog_manager.cc:1370] Generated new certificate authority record
Sep 25 21:58:33 dist-test-slave-2g0f krb5kdc[15909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:33.080981 16002 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:58:33.084900 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:33.074264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.2.70.254:33299)
0925 21:58:33.074420 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:33.074429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:33.074463 (+ 34us) server_negotiation.cc:407] Connection header received
0925 21:58:33.074575 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:33.074582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:33.074639 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:33.074787 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:33.075391 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:33.076840 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:33.077893 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:33.078565 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:33.078677 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:33.081373 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:33.081378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:33.081379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:33.081391 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:33.083311 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:33.083687 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:33.083689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:33.083690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:33.083801 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:33.084018 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:33.084019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:33.084019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:33.084301 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:33.084450 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:33.084518 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 21:58:33.084668 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260925 21:58:33.086131 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 21:58:33.086380 16002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4758f38482394ede98f4fc463a7287cc: Generated new TSK 0
I20260925 21:58:33.086598 16002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:58:33.088837 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 15922
2026-09-25T21:58:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1208 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:58:33 dist-test-slave-2g0f krb5kdc[16017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:58:33 dist-test-slave-2g0f krb5kdc[16017](info): set up 2 sockets
Sep 25 21:58:33 dist-test-slave-2g0f krb5kdc[16017](info): commencing operation
krb5kdc: starting...
W20260925 21:58:34.035742 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:58:34 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:58:34.154625 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:46519
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:46519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:34.233968 16030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:34.234195 16030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:34.234218 16030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:34.235419 16030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:58:34.235466 16030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:34.235478 16030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:34.235491 16030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:58:34.235502 16030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:58:34.237157 16030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:46519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:46519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:34.237406 16030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:34.237619 16030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:34.242260 16036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:34.242264 16038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:34.242265 16035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:34.242496 16030 server_base.cc:1034] running on GCE node
I20260925 21:58:34.242710 16030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:34.242960 16030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:34.244117 16030 hybrid_clock.cc:648] HybridClock initialized: now 1790373514244093 us; error 32 us; skew 500 ppm
Sep 25 21:58:34 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373514, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:34.246629 16030 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:58:34.247100 16030 webserver.cc:460] Webserver started at http://127.2.70.254:33963/ using document root <none> and password file <none>
I20260925 21:58:34.247303 16030 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:34.247346 16030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:34.247450 16030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:58:34.248592 16030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "9073fed4836b48b9b0bf8154a09dbb49"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "0d9b68d9efcc3ec5207156474573e55f"
server_key_iv: "4033802b6a467c2826032e3b30ddc7ea"
server_key_version: "encryptionkey@0"
I20260925 21:58:34.248919 16030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "9073fed4836b48b9b0bf8154a09dbb49"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "0d9b68d9efcc3ec5207156474573e55f"
server_key_iv: "4033802b6a467c2826032e3b30ddc7ea"
server_key_version: "encryptionkey@0"
I20260925 21:58:34.250188 16030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:58:34.250950 16045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:34.251142 16030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:34.251226 16030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "9073fed4836b48b9b0bf8154a09dbb49"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "0d9b68d9efcc3ec5207156474573e55f"
server_key_iv: "4033802b6a467c2826032e3b30ddc7ea"
server_key_version: "encryptionkey@0"
I20260925 21:58:34.251288 16030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:34.272198 16030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:34.274410 16030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:34.274541 16030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:34.278841 16030 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:46519
I20260925 21:58:34.278944 16097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:46519 every 8 connection(s)
I20260925 21:58:34.279277 16030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:58:34.280365 16098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:34.282655 16098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Bootstrap starting.
I20260925 21:58:34.283237 16098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:34.283496 16098 log.cc:824] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:34.284258 16098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: No bootstrap required, opened a new log
I20260925 21:58:34.285838 16098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } }
I20260925 21:58:34.285995 16098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:34.286026 16098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9073fed4836b48b9b0bf8154a09dbb49, State: Initialized, Role: FOLLOWER
I20260925 21:58:34.286182 16098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } }
I20260925 21:58:34.286264 16098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:58:34.286298 16098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:58:34.286346 16098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:34.287153 16098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } }
I20260925 21:58:34.287252 16098 leader_election.cc:302] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9073fed4836b48b9b0bf8154a09dbb49; no voters:
I20260925 21:58:34.287412 16098 leader_election.cc:288] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:58:34.287568 16102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:58:34.287673 16098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:58:34.287695 16102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 1 LEADER]: Becoming Leader. State: Replica: 9073fed4836b48b9b0bf8154a09dbb49, State: Running, Role: LEADER
I20260925 21:58:34.287770 16102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } }
I20260925 21:58:34.288156 16103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9073fed4836b48b9b0bf8154a09dbb49. Latest consensus state: current_term: 1 leader_uuid: "9073fed4836b48b9b0bf8154a09dbb49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } } }
I20260925 21:58:34.288174 16101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9073fed4836b48b9b0bf8154a09dbb49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } } }
I20260925 21:58:34.288309 16103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:34.288327 16101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:34.288829 16110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:58:34.289485 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 16030
I20260925 21:58:34.289577 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:58:34.289353 16110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:58:34.289762 2331 external_mini_cluster.cc:1428] Setting key 27b142f3c5e614ef0a5b7c6d6f59cf75
I20260925 21:58:34.291373 16110 catalog_manager.cc:1347] Generated new cluster ID: 5d2d9093cb9b4e038f6e5d1916867da5
I20260925 21:58:34.291424 16110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 21:58:34 dist-test-slave-2g0f krb5kdc[16017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:34.299571 16110 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:58:34.300154 16110 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:58:34.301720 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.290243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.2.70.254:46519)
0925 21:58:34.290418 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:34.290428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:34.290865 (+ 437us) server_negotiation.cc:407] Connection header received
0925 21:58:34.291020 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:34.291028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:34.291086 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:34.291234 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:34.292022 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.293430 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.294521 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.295156 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.295248 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.298046 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:34.298050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:34.298051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:34.298064 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:34.300059 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:34.300494 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:34.300498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:34.300498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:34.300622 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:34.300795 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:34.300795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:34.300796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:34.301080 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:34.301207 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:34.301308 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 21:58:34.301459 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260925 21:58:34.317054 16110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Generated new TSK 0
I20260925 21:58:34.317274 16110 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:58:34.351246 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46519
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:34.433245 16124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:34.433476 16124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:34.433499 16124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:34.434710 16124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:34.434759 16124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:34.434787 16124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:58:34.436349 16124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:34.436592 16124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:34.436820 16124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:58:34.441864 16124 server_base.cc:1034] running on GCE node
W20260925 21:58:34.441803 16130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:34.441807 16132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:34.441830 16129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:34.442255 16124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:34.442523 16124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:34.443675 16124 hybrid_clock.cc:648] HybridClock initialized: now 1790373514443663 us; error 29 us; skew 500 ppm
Sep 25 21:58:34 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373514, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:34.446336 16124 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:58:34.446787 16124 webserver.cc:460] Webserver started at http://127.2.70.193:36669/ using document root <none> and password file <none>
I20260925 21:58:34.446992 16124 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:34.447042 16124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:34.447168 16124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:58:34.448295 16124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "48653e5edbba40d8a5c7b4e67163a33b"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "26469f8aa2fa5bef7ff8b7e7b647220e"
server_key_iv: "9f924744dcd1b6d00bd4548b79c17f27"
server_key_version: "encryptionkey@0"
I20260925 21:58:34.448630 16124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "48653e5edbba40d8a5c7b4e67163a33b"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "26469f8aa2fa5bef7ff8b7e7b647220e"
server_key_iv: "9f924744dcd1b6d00bd4548b79c17f27"
server_key_version: "encryptionkey@0"
I20260925 21:58:34.449865 16124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:34.450647 16139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:34.450809 16124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:34.450892 16124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "48653e5edbba40d8a5c7b4e67163a33b"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "26469f8aa2fa5bef7ff8b7e7b647220e"
server_key_iv: "9f924744dcd1b6d00bd4548b79c17f27"
server_key_version: "encryptionkey@0"
I20260925 21:58:34.450954 16124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:34.463011 16124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:34.465056 16124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:34.465199 16124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:34.465422 16124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:34.465720 16124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:34.465752 16124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:34.465782 16124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:34.465799 16124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:34.471155 16124 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:39871
I20260925 21:58:34.471220 16252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:39871 every 8 connection(s)
I20260925 21:58:34.471513 16124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:58:34.476097 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 16124
I20260925 21:58:34.476186 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:58:34.476393 2331 external_mini_cluster.cc:1428] Setting key 0c6cb5a088d071c555d29dcd9c6d0824
Sep 25 21:58:34 dist-test-slave-2g0f krb5kdc[16017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373514, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:34.481177 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.472239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:50965 (local address 127.2.70.254:46519)
0925 21:58:34.472318 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:34.472321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:34.472454 (+ 133us) server_negotiation.cc:407] Connection header received
0925 21:58:34.472972 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:34.472973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:34.472983 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:34.473034 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:34.473974 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.474479 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.475450 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.475659 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.475713 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.478030 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:34.478031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:34.478032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:34.478039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:34.479584 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:34.480107 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:34.480108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:34.480108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:34.480171 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:34.480431 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:34.480432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:34.480432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:34.480559 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:34.480626 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:34.481018 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 21:58:34.481109 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 21:58:34.481379 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.472196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:50965)
0925 21:58:34.472335 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:34.472373 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:34.472861 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:34.473113 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:34.473117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:34.473708 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:34.473901 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.473906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.474555 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.474555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.475359 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.475361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.475720 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.475721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.475815 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.476369 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:34.476377 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:34.477937 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:34.479678 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:34.479690 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:34.479693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:34.480017 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:34.480235 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:34.480236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:34.480237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:34.480370 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:34.480623 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:34.480630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:34.480809 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 21:58:34.480960 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 21:58:34.481145 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260925 21:58:34.481840 16253 heartbeater.cc:347] Connected to a master server at 127.2.70.254:46519
I20260925 21:58:34.481937 16253 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:34.482172 16253 heartbeater.cc:511] Master 127.2.70.254:46519 requested a full tablet report, sending...
I20260925 21:58:34.482733 16062 ts_manager.cc:194] Registered new tserver with Master: 48653e5edbba40d8a5c7b4e67163a33b (127.2.70.193:39871)
I20260925 21:58:34.483625 16062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:50965
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:58:34.503291 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46519
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:34.591257 16258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:34.591483 16258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:34.591508 16258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:34.592733 16258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:34.592772 16258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:34.592797 16258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:58:34.594285 16258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:34.594491 16258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:34.594681 16258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:34.599507 16263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:34.599520 16264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:34.599502 16266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:34.599622 16258 server_base.cc:1034] running on GCE node
I20260925 21:58:34.599983 16258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:34.600214 16258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:34.601368 16258 hybrid_clock.cc:648] HybridClock initialized: now 1790373514601349 us; error 34 us; skew 500 ppm
Sep 25 21:58:34 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373514, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:34.603942 16258 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:58:34.604372 16258 webserver.cc:460] Webserver started at http://127.2.70.194:42351/ using document root <none> and password file <none>
I20260925 21:58:34.604583 16258 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:34.604631 16258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:34.604756 16258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:58:34.605829 16258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "84a228ad870e4dc2b264803b3fbf5675"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "66d1cac6897afbffd987352cb8ac02e5"
server_key_iv: "e5b1ed405ab7f7bbbe447c165f177669"
server_key_version: "encryptionkey@0"
I20260925 21:58:34.606158 16258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "84a228ad870e4dc2b264803b3fbf5675"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "66d1cac6897afbffd987352cb8ac02e5"
server_key_iv: "e5b1ed405ab7f7bbbe447c165f177669"
server_key_version: "encryptionkey@0"
I20260925 21:58:34.607367 16258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:58:34.608053 16273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:34.608196 16258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:58:34.608285 16258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "84a228ad870e4dc2b264803b3fbf5675"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "66d1cac6897afbffd987352cb8ac02e5"
server_key_iv: "e5b1ed405ab7f7bbbe447c165f177669"
server_key_version: "encryptionkey@0"
I20260925 21:58:34.608345 16258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:34.626412 16258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:34.628237 16258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:34.628366 16258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:34.628590 16258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:34.628880 16258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:34.628916 16258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:34.628947 16258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:34.628963 16258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:34.634411 16258 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:38947
I20260925 21:58:34.634452 16386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:38947 every 8 connection(s)
I20260925 21:58:34.634752 16258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:58:34.635560 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 16258
I20260925 21:58:34.635653 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:58:34.635856 2331 external_mini_cluster.cc:1428] Setting key 4cfbe0eca350d1d5f3ad1f06928628cf
Sep 25 21:58:34 dist-test-slave-2g0f krb5kdc[16017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373514, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
I20260925 21:58:34.644297 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.635445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:34841 (local address 127.2.70.254:46519)
0925 21:58:34.635508 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:34.635510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:34.635730 (+ 220us) server_negotiation.cc:407] Connection header received
0925 21:58:34.636360 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:34.636362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:34.636374 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:34.636427 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:34.637646 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.638357 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.639470 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.639670 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.639726 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.641785 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:34.641786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:34.641787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:34.641795 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:34.643053 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:34.643406 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:34.643407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:34.643407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:34.643466 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:34.643681 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:34.643682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:34.643682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:34.643790 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:34.643862 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:34.644122 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 21:58:34.644205 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 21:58:34.644465 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.635412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:34841)
0925 21:58:34.635569 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:34.635654 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:34.636271 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:34.636499 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:34.636503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:34.637260 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:34.637531 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.637538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.638454 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.638455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.639390 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.639393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.639738 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.639739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.639902 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.640407 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:34.640415 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:34.641699 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:34.643124 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:34.643131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:34.643133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:34.643342 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:34.643523 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:34.643524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:34.643524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:34.643628 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:34.643861 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:34.643867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:34.644000 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 21:58:34.644096 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 21:58:34.644250 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260925 21:58:34.644881 16387 heartbeater.cc:347] Connected to a master server at 127.2.70.254:46519
I20260925 21:58:34.644961 16387 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:34.645146 16387 heartbeater.cc:511] Master 127.2.70.254:46519 requested a full tablet report, sending...
I20260925 21:58:34.645515 16062 ts_manager.cc:194] Registered new tserver with Master: 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947)
I20260925 21:58:34.645963 16062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:34841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:58:34.662300 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46519
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:34.741379 16392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:34.741585 16392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:34.741606 16392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:34.742806 16392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:34.742856 16392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:34.742887 16392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:58:34.744361 16392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:34.744585 16392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:34.744791 16392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:34.749660 16398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:34.749682 16400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:34.749737 16392 server_base.cc:1034] running on GCE node
W20260925 21:58:34.749701 16397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:34.750048 16392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:34.750290 16392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:34.751487 16392 hybrid_clock.cc:648] HybridClock initialized: now 1790373514751471 us; error 25 us; skew 500 ppm
Sep 25 21:58:34 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373514, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:34.754107 16392 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:58:34.754621 16392 webserver.cc:460] Webserver started at http://127.2.70.195:44889/ using document root <none> and password file <none>
I20260925 21:58:34.754891 16392 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:34.754998 16392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:34.755131 16392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:58:34.756219 16392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "09e95fb1bd9349dcbab485f623d7301f"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "e69630fa80086d289d3857098466e4a8"
server_key_iv: "031aba9fd9b7d885fd3ebc284b9594f3"
server_key_version: "encryptionkey@0"
I20260925 21:58:34.756618 16392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "09e95fb1bd9349dcbab485f623d7301f"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "e69630fa80086d289d3857098466e4a8"
server_key_iv: "031aba9fd9b7d885fd3ebc284b9594f3"
server_key_version: "encryptionkey@0"
I20260925 21:58:34.757854 16392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:34.758594 16407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:34.758793 16392 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:58:34.758872 16392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "09e95fb1bd9349dcbab485f623d7301f"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "e69630fa80086d289d3857098466e4a8"
server_key_iv: "031aba9fd9b7d885fd3ebc284b9594f3"
server_key_version: "encryptionkey@0"
I20260925 21:58:34.758951 16392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:34.776048 16392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:34.778604 16392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:34.778759 16392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:34.779028 16392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:34.779347 16392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:34.779381 16392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:34.779402 16392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:34.779423 16392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:34.785040 16392 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:38705
I20260925 21:58:34.785079 16520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:38705 every 8 connection(s)
I20260925 21:58:34.785388 16392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:58:34.787099 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 16392
I20260925 21:58:34.787205 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:58:34.787391 2331 external_mini_cluster.cc:1428] Setting key ccbc1ad0aa224702b7127d23ae4cce82
Sep 25 21:58:34 dist-test-slave-2g0f krb5kdc[16017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373514, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:34.794873 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.786076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:45341 (local address 127.2.70.254:46519)
0925 21:58:34.786137 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:34.786139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:34.786301 (+ 162us) server_negotiation.cc:407] Connection header received
0925 21:58:34.786820 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:34.786822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:34.786833 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:34.786904 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:34.788225 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.788975 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.790046 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.790251 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.790315 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.792516 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:34.792518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:34.792518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:34.792525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:34.793695 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:34.794059 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:34.794060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:34.794061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:34.794119 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:34.794345 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:34.794346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:34.794346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:34.794474 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:34.794532 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:34.794746 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 21:58:34.794806 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 21:58:34.795089 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.786038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:45341)
0925 21:58:34.786184 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:34.786225 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:34.786709 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:34.787009 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:34.787014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:34.787811 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:34.788115 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.788123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.789091 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.789092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.789954 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.789956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.790325 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.790326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.790436 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.790965 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:34.790974 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:34.792429 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:34.793765 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:34.793771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:34.793773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:34.793980 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:34.794179 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:34.794179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:34.794180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:34.794274 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:34.794527 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:34.794531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:34.794646 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 21:58:34.794724 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 21:58:34.794842 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260925 21:58:34.795538 16521 heartbeater.cc:347] Connected to a master server at 127.2.70.254:46519
I20260925 21:58:34.795639 16521 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:34.795814 16521 heartbeater.cc:511] Master 127.2.70.254:46519 requested a full tablet report, sending...
I20260925 21:58:34.796274 16062 ts_manager.cc:194] Registered new tserver with Master: 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705)
I20260925 21:58:34.796626 16062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:45341
I20260925 21:58:34.799316 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:58:34.806354 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.800621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.2.70.254:46519)
0925 21:58:34.800670 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:34.800672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:34.800777 (+ 105us) server_negotiation.cc:407] Connection header received
0925 21:58:34.800821 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:34.800822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:34.800830 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:34.800870 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:34.801396 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.801897 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.802605 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.802788 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.802842 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.803999 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:34.804000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:34.804000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:34.804007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:34.805338 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:34.805644 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:34.805644 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:34.805645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:34.805702 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:34.805829 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:34.805829 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:34.805829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:34.806015 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:34.806090 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:34.806233 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 21:58:34.806289 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 21:58:34.807823 16062 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:58:34.809365 16062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:58:34.815673 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.811621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.2.70.193:39871)
0925 21:58:34.811782 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:34.811785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:34.812161 (+ 376us) server_negotiation.cc:407] Connection header received
0925 21:58:34.812202 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:34.812205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:34.812218 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:34.812282 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:34.812402 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.813510 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.814842 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.815312 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.815372 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.815432 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:34.815559 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 21:58:34.815597 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260925 21:58:34.815665 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.811298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:39871 (local address 127.0.0.1:58026)
0925 21:58:34.812056 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:34.812064 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:34.812106 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:34.812292 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:34.812293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:34.812295 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:34.812342 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.812343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.813606 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.813607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.814774 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.814776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.815386 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.815387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.815460 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.815473 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 21:58:34.815537 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 21:58:34.815586 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":28,"thread_start_us":60,"threads_started":1}
I20260925 21:58:34.815991 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.811398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.2.70.195:38705)
0925 21:58:34.811804 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:34.811809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:34.811932 (+ 123us) server_negotiation.cc:407] Connection header received
0925 21:58:34.812062 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:34.812065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:34.812084 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:34.812376 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:34.812453 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.813241 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.814651 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.815358 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.815423 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.815478 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:34.815866 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 21:58:34.815906 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":114,"threads_started":1}
I20260925 21:58:34.816069 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.811297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:38705 (local address 127.0.0.1:43924)
0925 21:58:34.811794 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:34.811866 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:34.811938 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:34.812228 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:34.812231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:34.812233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:34.812305 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.812307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.813329 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.813330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.814567 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.814570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.815514 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.815515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.815586 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.815590 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:58:34.815634 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 21:58:34.815675 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":26,"spinlock_wait_cycles":384,"thread_start_us":68,"threads_started":1}
I20260925 21:58:34.816294 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.811607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.2.70.194:38947)
0925 21:58:34.811767 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:34.811772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:34.812433 (+ 661us) server_negotiation.cc:407] Connection header received
0925 21:58:34.812484 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:34.812487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:34.812502 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:34.812575 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:34.812681 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.813694 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.814794 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.815472 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.815525 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.815574 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:34.816027 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 21:58:34.816223 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260925 21:58:34.816435 16187 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd0f9d61c56434da243797ad7d38bb0 (DEFAULT_TABLE table=test-table [id=e10a24d468d2442e949eab06bd4ca325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:58:34.816694 16455 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd0f9d61c56434da243797ad7d38bb0 (DEFAULT_TABLE table=test-table [id=e10a24d468d2442e949eab06bd4ca325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:58:34.816797 16187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd0f9d61c56434da243797ad7d38bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:34.816987 16455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd0f9d61c56434da243797ad7d38bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:34.819926 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.811483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:38947 (local address 127.0.0.1:57924)
0925 21:58:34.812374 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:34.812384 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:34.812432 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:34.812582 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:34.812582 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:34.812584 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:34.812625 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.812627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.813778 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.813779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.814706 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.814709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.815601 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.815601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.815652 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.815656 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:58:34.815693 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 21:58:34.815724 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":33,"threads_started":1}
I20260925 21:58:34.820290 16535 tablet_bootstrap.cc:492] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Bootstrap starting.
I20260925 21:58:34.820667 16321 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd0f9d61c56434da243797ad7d38bb0 (DEFAULT_TABLE table=test-table [id=e10a24d468d2442e949eab06bd4ca325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:58:34.820935 16535 tablet_bootstrap.cc:654] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:34.821000 16321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd0f9d61c56434da243797ad7d38bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:34.821471 16535 log.cc:824] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:34.821818 16536 tablet_bootstrap.cc:492] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f: Bootstrap starting.
I20260925 21:58:34.822235 16535 tablet_bootstrap.cc:492] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: No bootstrap required, opened a new log
I20260925 21:58:34.822296 16535 ts_tablet_manager.cc:1397] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:58:34.822460 16536 tablet_bootstrap.cc:654] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:34.822984 16536 log.cc:824] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:34.823743 16536 tablet_bootstrap.cc:492] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f: No bootstrap required, opened a new log
I20260925 21:58:34.823812 16536 ts_tablet_manager.cc:1397] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:58:34.824013 16535 raft_consensus.cc:348] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } }
I20260925 21:58:34.824198 16535 raft_consensus.cc:374] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:34.824239 16535 raft_consensus.cc:729] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48653e5edbba40d8a5c7b4e67163a33b, State: Initialized, Role: FOLLOWER
I20260925 21:58:34.824328 16539 tablet_bootstrap.cc:492] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675: Bootstrap starting.
I20260925 21:58:34.824364 16535 consensus_queue.cc:260] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } }
I20260925 21:58:34.824637 16535 ts_tablet_manager.cc:1428] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 21:58:34.824775 16253 heartbeater.cc:503] Master 127.2.70.254:46519 was elected leader, sending a full tablet report...
I20260925 21:58:34.824908 16539 tablet_bootstrap.cc:654] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:34.825266 16539 log.cc:824] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:34.825654 16536 raft_consensus.cc:348] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } }
I20260925 21:58:34.825784 16536 raft_consensus.cc:374] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:34.825814 16536 raft_consensus.cc:729] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e95fb1bd9349dcbab485f623d7301f, State: Initialized, Role: FOLLOWER
I20260925 21:58:34.825840 16539 tablet_bootstrap.cc:492] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675: No bootstrap required, opened a new log
I20260925 21:58:34.825883 16539 ts_tablet_manager.cc:1397] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:58:34.825922 16536 consensus_queue.cc:260] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } }
I20260925 21:58:34.826184 16536 ts_tablet_manager.cc:1428] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:58:34.826212 16521 heartbeater.cc:503] Master 127.2.70.254:46519 was elected leader, sending a full tablet report...
I20260925 21:58:34.827623 16539 raft_consensus.cc:348] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } }
I20260925 21:58:34.827755 16539 raft_consensus.cc:374] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:34.827776 16539 raft_consensus.cc:729] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a228ad870e4dc2b264803b3fbf5675, State: Initialized, Role: FOLLOWER
I20260925 21:58:34.827893 16539 consensus_queue.cc:260] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } }
I20260925 21:58:34.828094 16539 ts_tablet_manager.cc:1428] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:58:34.828148 16387 heartbeater.cc:503] Master 127.2.70.254:46519 was elected leader, sending a full tablet report...
W20260925 21:58:34.885288 16388 tablet.cc:2369] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:58:34.953476 16543 raft_consensus.cc:482] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:58:34.953639 16543 raft_consensus.cc:504] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } }
I20260925 21:58:34.954018 16543 leader_election.cc:288] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48653e5edbba40d8a5c7b4e67163a33b (127.2.70.193:39871), 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705)
I20260925 21:58:34.957603 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.954314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:51507 (local address 127.2.70.195:38705)
0925 21:58:34.954378 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:34.954380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:34.954532 (+ 152us) server_negotiation.cc:407] Connection header received
0925 21:58:34.954569 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:34.954570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:34.954579 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:34.954651 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:34.954790 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.955316 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.956663 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.957300 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.957364 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.957371 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:34.957492 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 21:58:34.957532 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 21:58:34.957603 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.954289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:36689 (local address 127.2.70.193:39871)
0925 21:58:34.954329 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:34.954331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:34.954343 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:58:34.954434 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:34.954435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:34.954445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:34.954522 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:34.954676 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.955263 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.956662 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.957238 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.957316 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.957326 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:34.957490 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 21:58:34.957531 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 21:58:34.957613 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.954255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:39871 (local address 127.2.70.194:36689)
0925 21:58:34.954298 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:34.954309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:34.954352 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:34.954536 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:34.954537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:34.954539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:34.954614 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.954615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.955335 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.955336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.956561 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.956564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.957327 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.957328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.957417 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.957422 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:58:34.957476 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 21:58:34.957518 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":6}
I20260925 21:58:34.957618 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.954255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:38705 (local address 127.2.70.194:51507)
0925 21:58:34.954459 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:34.954469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:34.954526 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:34.954662 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:34.954663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:34.954665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:34.954737 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.954739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.955375 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.955376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.956561 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:34.956564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.957375 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.957375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:34.957433 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.957435 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:58:34.957483 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 21:58:34.957517 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":49,"thread_start_us":95,"threads_started":1}
I20260925 21:58:34.958010 16207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd0f9d61c56434da243797ad7d38bb0" candidate_uuid: "84a228ad870e4dc2b264803b3fbf5675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48653e5edbba40d8a5c7b4e67163a33b" is_pre_election: true
I20260925 21:58:34.958186 16207 raft_consensus.cc:2454] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84a228ad870e4dc2b264803b3fbf5675 in term 0.
I20260925 21:58:34.958316 16475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd0f9d61c56434da243797ad7d38bb0" candidate_uuid: "84a228ad870e4dc2b264803b3fbf5675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e95fb1bd9349dcbab485f623d7301f" is_pre_election: true
I20260925 21:58:34.958420 16277 leader_election.cc:302] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48653e5edbba40d8a5c7b4e67163a33b, 84a228ad870e4dc2b264803b3fbf5675; no voters:
I20260925 21:58:34.958468 16475 raft_consensus.cc:2454] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84a228ad870e4dc2b264803b3fbf5675 in term 0.
I20260925 21:58:34.958537 16543 raft_consensus.cc:2781] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:58:34.958616 16543 raft_consensus.cc:482] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:58:34.958643 16543 raft_consensus.cc:3037] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:34.959333 16543 raft_consensus.cc:504] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } }
I20260925 21:58:34.959475 16543 leader_election.cc:288] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [CANDIDATE]: Term 1 election: Requested vote from peers 48653e5edbba40d8a5c7b4e67163a33b (127.2.70.193:39871), 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705)
I20260925 21:58:34.959640 16207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd0f9d61c56434da243797ad7d38bb0" candidate_uuid: "84a228ad870e4dc2b264803b3fbf5675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48653e5edbba40d8a5c7b4e67163a33b"
I20260925 21:58:34.959663 16475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd0f9d61c56434da243797ad7d38bb0" candidate_uuid: "84a228ad870e4dc2b264803b3fbf5675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e95fb1bd9349dcbab485f623d7301f"
I20260925 21:58:34.959723 16207 raft_consensus.cc:3037] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:34.959730 16475 raft_consensus.cc:3037] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:34.960608 16475 raft_consensus.cc:2454] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84a228ad870e4dc2b264803b3fbf5675 in term 1.
I20260925 21:58:34.960608 16207 raft_consensus.cc:2454] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84a228ad870e4dc2b264803b3fbf5675 in term 1.
I20260925 21:58:34.960805 16275 leader_election.cc:302] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09e95fb1bd9349dcbab485f623d7301f, 84a228ad870e4dc2b264803b3fbf5675; no voters:
I20260925 21:58:34.960883 16543 raft_consensus.cc:2781] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:58:34.961041 16543 raft_consensus.cc:686] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 1 LEADER]: Becoming Leader. State: Replica: 84a228ad870e4dc2b264803b3fbf5675, State: Running, Role: LEADER
I20260925 21:58:34.961128 16543 consensus_queue.cc:237] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } }
I20260925 21:58:34.961982 16061 catalog_manager.cc:5495] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194). New cstate: current_term: 1 leader_uuid: "84a228ad870e4dc2b264803b3fbf5675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } health_report { overall_health: HEALTHY } } }
W20260925 21:58:34.972249 16254 tablet.cc:2369] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:58:34.997678 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:34.994736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.2.70.194:38947)
0925 21:58:34.994866 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:34.994868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:34.994876 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:58:34.994911 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:34.994912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:34.994932 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:34.995000 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:58:34.995172 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.995683 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.996779 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:34.997061 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:34.997132 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:34.997294 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:58:34.997428 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:58:34.997511 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:34.997548 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 21:58:34.997570 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 21:58:34.999677 16475 raft_consensus.cc:1261] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f [term 1 FOLLOWER]: Refusing update from remote peer 84a228ad870e4dc2b264803b3fbf5675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:58:34.999677 16207 raft_consensus.cc:1261] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Refusing update from remote peer 84a228ad870e4dc2b264803b3fbf5675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:58:35.000069 16545 consensus_queue.cc:1035] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:58:35.000207 16543 consensus_queue.cc:1035] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:58:35.002655 16548 mvcc.cc:205] Tried to move back new op lower bound from 7333369917435801600 to 7333369917281308672. Current Snapshot: MvccSnapshot[applied={T|T < 7333369917435801600}]
I20260925 21:58:35.004573 16549 mvcc.cc:205] Tried to move back new op lower bound from 7333369917435801600 to 7333369917281308672. Current Snapshot: MvccSnapshot[applied={T|T < 7333369917435801600 or (T in {7333369917435801600})}]
I20260925 21:58:35.005256 16062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:58:35.005360 16062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:58:35.006157 16062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Sending DeleteTablet for 3 replicas of tablet 8fd0f9d61c56434da243797ad7d38bb0
I20260925 21:58:35.006485 16455 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd0f9d61c56434da243797ad7d38bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:35 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:43924
I20260925 21:58:35.006538 16187 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd0f9d61c56434da243797ad7d38bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:35 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:58026
I20260925 21:58:35.006738 16321 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd0f9d61c56434da243797ad7d38bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:35 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:57924
I20260925 21:58:35.006780 16559 tablet_replica.cc:330] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f: stopping tablet replica
I20260925 21:58:35.006883 16559 raft_consensus.cc:2229] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:58:35.006999 16559 raft_consensus.cc:2258] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:58:35.007009 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 16124
I20260925 21:58:35.007669 16561 tablet_replica.cc:330] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675: stopping tablet replica
I20260925 21:58:35.007773 16561 raft_consensus.cc:2229] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:58:35.007907 16561 raft_consensus.cc:2258] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:58:35.007920 16559 ts_tablet_manager.cc:1905] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:58:35.008307 16561 ts_tablet_manager.cc:1905] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:58:35.009862 16559 ts_tablet_manager.cc:1918] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:58:35.010010 16559 log.cc:1196] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/8fd0f9d61c56434da243797ad7d38bb0
I20260925 21:58:35.010018 16561 ts_tablet_manager.cc:1918] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:58:35.010090 16561 log.cc:1196] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/8fd0f9d61c56434da243797ad7d38bb0
I20260925 21:58:35.010368 16561 ts_tablet_manager.cc:1939] T 8fd0f9d61c56434da243797ad7d38bb0 P 84a228ad870e4dc2b264803b3fbf5675: Deleting consensus metadata
I20260925 21:58:35.010530 16559 ts_tablet_manager.cc:1939] T 8fd0f9d61c56434da243797ad7d38bb0 P 09e95fb1bd9349dcbab485f623d7301f: Deleting consensus metadata
I20260925 21:58:35.010886 16047 catalog_manager.cc:4841] TS 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947): tablet 8fd0f9d61c56434da243797ad7d38bb0 (table test-table [id=e10a24d468d2442e949eab06bd4ca325]) successfully deleted
I20260925 21:58:35.011073 16047 catalog_manager.cc:4841] TS 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705): tablet 8fd0f9d61c56434da243797ad7d38bb0 (table test-table [id=e10a24d468d2442e949eab06bd4ca325]) successfully deleted
I20260925 21:58:35.014101 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 16258
W20260925 21:58:35.016078 16049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:39871 (error 108)
I20260925 21:58:35.016640 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.016484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:39871 (local address 127.0.0.1:58032)
0925 21:58:35.016529 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:35.016600 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:39871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 21:58:35.016772 16049 catalog_manager.cc:4584] TS 48653e5edbba40d8a5c7b4e67163a33b (127.2.70.193:39871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fd0f9d61c56434da243797ad7d38bb0: Network error: Client connection negotiation failed: client connection to 127.2.70.193:39871: connect: Connection refused (error 111)
I20260925 21:58:35.019153 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 16392
I20260925 21:58:35.023715 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 16030
I20260925 21:58:35.028394 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:46519
--webserver_interface=127.2.70.254
--webserver_port=33963
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:46519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:35.109095 16563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:35.109315 16563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:35.109338 16563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:35.110535 16563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:58:35.110574 16563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:35.110605 16563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:35.110617 16563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:58:35.110634 16563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:58:35.112244 16563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:46519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:46519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:35.112555 16563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:35.112818 16563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:35.121132 16569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:35.121136 16572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:35.121186 16563 server_base.cc:1034] running on GCE node
W20260925 21:58:35.121204 16570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:35.124092 16563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:35.124426 16563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:35.125578 16563 hybrid_clock.cc:648] HybridClock initialized: now 1790373515125557 us; error 36 us; skew 500 ppm
Sep 25 21:58:35 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373515, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:35.128489 16563 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:58:35.128968 16563 webserver.cc:460] Webserver started at http://127.2.70.254:33963/ using document root <none> and password file <none>
I20260925 21:58:35.129186 16563 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:35.129249 16563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:35.130620 16563 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:35.131196 16579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:35.131343 16563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:58:35.131429 16563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "9073fed4836b48b9b0bf8154a09dbb49"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "0d9b68d9efcc3ec5207156474573e55f"
server_key_iv: "4033802b6a467c2826032e3b30ddc7ea"
server_key_version: "encryptionkey@0"
I20260925 21:58:35.131860 16563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:35.158891 16563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:35.160889 16563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:35.161038 16563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:35.164918 16563 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:46519
I20260925 21:58:35.164994 16631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:46519 every 8 connection(s)
I20260925 21:58:35.165343 16563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:58:35.167007 16632 sys_catalog.cc:263] Verifying existing consensus state
I20260925 21:58:35.167668 16632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Bootstrap starting.
I20260925 21:58:35.174081 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 16563
I20260925 21:58:35.174340 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:39871
--local_ip_for_outbound_sockets=127.2.70.193
--tserver_master_addrs=127.2.70.254:46519
--webserver_port=36669
--webserver_interface=127.2.70.193
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:58:35.175704 16632 log.cc:824] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:35.177572 16632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 21:58:35.177865 16632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Bootstrap complete.
I20260925 21:58:35.180063 16632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } }
I20260925 21:58:35.180296 16632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9073fed4836b48b9b0bf8154a09dbb49, State: Initialized, Role: FOLLOWER
I20260925 21:58:35.180441 16632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } }
I20260925 21:58:35.180505 16632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:58:35.180550 16632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:58:35.180598 16632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 1 FOLLOWER]: Advancing to term 2
I20260925 21:58:35.181638 16632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } }
I20260925 21:58:35.181728 16632 leader_election.cc:302] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9073fed4836b48b9b0bf8154a09dbb49; no voters:
I20260925 21:58:35.181897 16632 leader_election.cc:288] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 21:58:35.181932 16636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 21:58:35.182040 16636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 2 LEADER]: Becoming Leader. State: Replica: 9073fed4836b48b9b0bf8154a09dbb49, State: Running, Role: LEADER
I20260925 21:58:35.182140 16636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } }
I20260925 21:58:35.182183 16632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:58:35.182415 16638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9073fed4836b48b9b0bf8154a09dbb49. Latest consensus state: current_term: 2 leader_uuid: "9073fed4836b48b9b0bf8154a09dbb49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } } }
I20260925 21:58:35.182673 16638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:35.182608 16637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9073fed4836b48b9b0bf8154a09dbb49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } } }
I20260925 21:58:35.182934 16637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:35.182981 16646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:58:35.183748 16646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:58:35.183867 16646 catalog_manager.cc:1259] Loaded cluster ID: 5d2d9093cb9b4e038f6e5d1916867da5
I20260925 21:58:35.183899 16646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:58:35.184119 16646 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 21:58:35.188918 16646 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 21:58:35.192005 16646 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:58:35.192243 16646 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Loaded TSK: 0
I20260925 21:58:35.192479 16646 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 21:58:35.257556 16634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:35.257781 16634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:35.257817 16634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:35.259079 16634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:35.259128 16634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:35.259166 16634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:58:35.260740 16634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:39871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=36669
--tserver_master_addrs=127.2.70.254:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:35.260984 16634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:35.261211 16634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:35.267446 16660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:35.267443 16659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:35.267567 16662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:35.267464 16634 server_base.cc:1034] running on GCE node
I20260925 21:58:35.268031 16634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:35.268276 16634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:35.269438 16634 hybrid_clock.cc:648] HybridClock initialized: now 1790373515269412 us; error 39 us; skew 500 ppm
Sep 25 21:58:35 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373515, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:35.272164 16634 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:58:35.272660 16634 webserver.cc:460] Webserver started at http://127.2.70.193:36669/ using document root <none> and password file <none>
I20260925 21:58:35.272884 16634 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:35.272949 16634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:35.274289 16634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 21:58:35.274912 16669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:35.275100 16634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:35.275209 16634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "48653e5edbba40d8a5c7b4e67163a33b"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "26469f8aa2fa5bef7ff8b7e7b647220e"
server_key_iv: "9f924744dcd1b6d00bd4548b79c17f27"
server_key_version: "encryptionkey@0"
I20260925 21:58:35.275694 16634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:35.288239 16634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:35.290923 16634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:35.291095 16634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:35.291393 16634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:35.291952 16676 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 21:58:35.293469 16634 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 21:58:35.293536 16634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 21:58:35.293576 16634 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 21:58:35.294504 16634 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 21:58:35.294557 16634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 21:58:35.294627 16676 tablet_bootstrap.cc:492] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Bootstrap starting.
I20260925 21:58:35.302399 16634 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:39871
I20260925 21:58:35.302879 16634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:58:35.304868 16676 log.cc:824] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:35.306836 16676 tablet_bootstrap.cc:492] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 21:58:35.307090 16676 tablet_bootstrap.cc:492] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Bootstrap complete.
I20260925 21:58:35.307816 16676 ts_tablet_manager.cc:1397] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260925 21:58:35.308069 16783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:39871 every 8 connection(s)
I20260925 21:58:35.308965 16676 raft_consensus.cc:348] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } }
I20260925 21:58:35.309273 16676 raft_consensus.cc:729] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48653e5edbba40d8a5c7b4e67163a33b, State: Initialized, Role: FOLLOWER
I20260925 21:58:35.309348 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 16634
I20260925 21:58:35.309425 16676 consensus_queue.cc:260] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } }
I20260925 21:58:35.309605 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:38947
--local_ip_for_outbound_sockets=127.2.70.194
--tserver_master_addrs=127.2.70.254:46519
--webserver_port=42351
--webserver_interface=127.2.70.194
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:58:35.309679 16676 ts_tablet_manager.cc:1428] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 25 21:58:35 dist-test-slave-2g0f krb5kdc[16017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373515, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:35.320173 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.303694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:34377 (local address 127.2.70.254:46519)
0925 21:58:35.308416 (+ 4722us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:35.308427 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:35.308475 (+ 48us) server_negotiation.cc:407] Connection header received
0925 21:58:35.308618 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:35.308625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:35.308685 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:35.308854 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:35.309960 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.310922 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.312473 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.313107 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.313222 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.315974 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:35.315979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:35.315980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:35.315994 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:35.318098 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:35.318618 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:35.318622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:35.318622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:35.318745 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:35.319048 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:35.319049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:35.319049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:35.319356 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:35.319450 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:35.319813 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 21:58:35.319975 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4652,"thread_start_us":88,"threads_started":1}
I20260925 21:58:35.320180 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.303579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:34377)
0925 21:58:35.303767 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:35.303813 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:35.304520 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:35.308901 (+ 4381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:35.308906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:35.309298 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:35.309589 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.309596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.311076 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.311078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.312364 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.312368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.313221 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.313223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.313373 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.313995 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:35.314005 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:35.315792 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:35.318191 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:35.318199 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:35.318201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:35.318517 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:35.318820 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:35.318820 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:35.318821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:35.318964 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:35.319442 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:35.319450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:35.319632 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 21:58:35.319747 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 21:58:35.319962 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260925 21:58:35.321120 16784 heartbeater.cc:347] Connected to a master server at 127.2.70.254:46519
I20260925 21:58:35.321226 16784 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:35.321412 16784 heartbeater.cc:511] Master 127.2.70.254:46519 requested a full tablet report, sending...
I20260925 21:58:35.322144 16596 ts_manager.cc:194] Registered new tserver with Master: 48653e5edbba40d8a5c7b4e67163a33b (127.2.70.193:39871)
I20260925 21:58:35.323342 16596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:34377
I20260925 21:58:35.324330 16784 heartbeater.cc:503] Master 127.2.70.254:46519 was elected leader, sending a full tablet report...
I20260925 21:58:35.331390 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.322747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:39871 (local address 127.0.0.1:58046)
0925 21:58:35.327853 (+ 5106us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:35.327894 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:35.327973 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:35.328160 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:35.328162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:35.328164 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:35.328217 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.328219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.329422 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.329423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.330274 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.330277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.331139 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.331139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.331217 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.331227 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 21:58:35.331284 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 21:58:35.331326 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5054,"thread_start_us":98,"threads_started":1}
I20260925 21:58:35.331426 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.323623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.2.70.193:39871)
0925 21:58:35.323921 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:35.323925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:35.327972 (+ 4047us) server_negotiation.cc:407] Connection header received
0925 21:58:35.328047 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:35.328049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:35.328067 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:35.328151 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:35.328264 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.329318 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.330361 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.331052 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.331126 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.331179 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:35.331306 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 21:58:35.331347 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":76,"threads_started":1}
I20260925 21:58:35.331787 16718 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd0f9d61c56434da243797ad7d38bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:35 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:58046
I20260925 21:58:35.332316 16793 tablet_replica.cc:330] stopping tablet replica
I20260925 21:58:35.332427 16793 raft_consensus.cc:2229] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:58:35.332495 16793 raft_consensus.cc:2258] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:58:35.332798 16793 ts_tablet_manager.cc:1905] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:58:35.335318 16793 ts_tablet_manager.cc:1918] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:58:35.335394 16793 log.cc:1196] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal/wals/8fd0f9d61c56434da243797ad7d38bb0
I20260925 21:58:35.335606 16793 ts_tablet_manager.cc:1939] T 8fd0f9d61c56434da243797ad7d38bb0 P 48653e5edbba40d8a5c7b4e67163a33b: Deleting consensus metadata
I20260925 21:58:35.336120 16583 catalog_manager.cc:4841] TS 48653e5edbba40d8a5c7b4e67163a33b (127.2.70.193:39871): tablet 8fd0f9d61c56434da243797ad7d38bb0 (table test-table [id=e10a24d468d2442e949eab06bd4ca325]) successfully deleted
W20260925 21:58:35.395761 16790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:35.396003 16790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:35.396028 16790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:35.397244 16790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:35.397291 16790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:35.397317 16790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:58:35.398782 16790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:38947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=42351
--tserver_master_addrs=127.2.70.254:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:35.399004 16790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:35.399225 16790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:58:35.404490 16790 server_base.cc:1034] running on GCE node
W20260925 21:58:35.404405 16799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:35.404436 16798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:35.404431 16801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:35.404801 16790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:35.405042 16790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:35.406193 16790 hybrid_clock.cc:648] HybridClock initialized: now 1790373515406182 us; error 27 us; skew 500 ppm
Sep 25 21:58:35 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373515, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:35.412071 16790 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:58:35.412593 16790 webserver.cc:460] Webserver started at http://127.2.70.194:42351/ using document root <none> and password file <none>
I20260925 21:58:35.412806 16790 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:35.412871 16790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:35.414279 16790 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:35.414870 16808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:35.415037 16790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:58:35.415124 16790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "84a228ad870e4dc2b264803b3fbf5675"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "66d1cac6897afbffd987352cb8ac02e5"
server_key_iv: "e5b1ed405ab7f7bbbe447c165f177669"
server_key_version: "encryptionkey@0"
I20260925 21:58:35.415488 16790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:35.442936 16790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:35.445003 16790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:35.445147 16790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:35.445374 16790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:35.445674 16790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:35.445706 16790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:35.445736 16790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:35.445760 16790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:35.451454 16790 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:38947
I20260925 21:58:35.451500 16921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:38947 every 8 connection(s)
I20260925 21:58:35.451822 16790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:58:35.455080 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 16790
I20260925 21:58:35.455319 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:38705
--local_ip_for_outbound_sockets=127.2.70.195
--tserver_master_addrs=127.2.70.254:46519
--webserver_port=44889
--webserver_interface=127.2.70.195
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 21:58:35 dist-test-slave-2g0f krb5kdc[16017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373515, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:35.461328 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.452515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:38481 (local address 127.2.70.254:46519)
0925 21:58:35.452557 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:35.452558 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:35.452764 (+ 206us) server_negotiation.cc:407] Connection header received
0925 21:58:35.453215 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:35.453216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:35.453229 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:35.453276 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:35.454171 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.454666 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.455686 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.455959 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.456019 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.458438 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:35.458440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:35.458440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:35.458449 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:35.460005 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:35.460406 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:35.460407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:35.460408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:35.460469 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:35.460677 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:35.460677 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:35.460678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:35.460816 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:35.460884 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:35.461135 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 21:58:35.461228 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 21:58:35.461488 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.452509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:38481)
0925 21:58:35.452640 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:35.452674 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:35.453130 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:35.453335 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:35.453338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:35.453887 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:35.454081 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.454086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.454733 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.454733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.455608 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.455610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.456028 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.456029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.456151 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.456648 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:35.456658 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:35.458326 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:35.460094 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:35.460100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:35.460101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:35.460328 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:35.460531 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:35.460532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:35.460532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:35.460626 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:35.460884 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:35.460890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:35.461017 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 21:58:35.461097 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 21:58:35.461218 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260925 21:58:35.462006 16922 heartbeater.cc:347] Connected to a master server at 127.2.70.254:46519
I20260925 21:58:35.462123 16922 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:35.462332 16922 heartbeater.cc:511] Master 127.2.70.254:46519 requested a full tablet report, sending...
I20260925 21:58:35.462815 16596 ts_manager.cc:194] Registered new tserver with Master: 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947)
I20260925 21:58:35.463176 16596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:38481
W20260925 21:58:35.537565 16925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:35.537775 16925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:35.537799 16925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:35.539065 16925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:35.539114 16925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:35.539140 16925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:58:35.540661 16925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:38705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=44889
--tserver_master_addrs=127.2.70.254:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:35.540972 16925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:35.541203 16925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:35.546151 16931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:35.546222 16930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:35.546207 16925 server_base.cc:1034] running on GCE node
W20260925 21:58:35.546162 16933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:35.546576 16925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:35.546797 16925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:35.547924 16925 hybrid_clock.cc:648] HybridClock initialized: now 1790373515547889 us; error 51 us; skew 500 ppm
Sep 25 21:58:35 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373515, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:35.550397 16925 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:58:35.550827 16925 webserver.cc:460] Webserver started at http://127.2.70.195:44889/ using document root <none> and password file <none>
I20260925 21:58:35.551033 16925 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:35.551096 16925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:35.552508 16925 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:35.553020 16940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:35.553189 16925 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:58:35.553270 16925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "09e95fb1bd9349dcbab485f623d7301f"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "e69630fa80086d289d3857098466e4a8"
server_key_iv: "031aba9fd9b7d885fd3ebc284b9594f3"
server_key_version: "encryptionkey@0"
I20260925 21:58:35.553606 16925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:35.586501 16925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:35.588557 16925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:35.588694 16925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:35.588917 16925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:35.589212 16925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:35.589244 16925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:35.589274 16925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:35.589293 16925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:35.594659 16925 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:38705
I20260925 21:58:35.594735 17053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:38705 every 8 connection(s)
I20260925 21:58:35.595034 16925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:58:35.600808 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 16925
Sep 25 21:58:35 dist-test-slave-2g0f krb5kdc[16017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373515, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:35.604353 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.595690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:45239 (local address 127.2.70.254:46519)
0925 21:58:35.595744 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:35.595747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:35.595986 (+ 239us) server_negotiation.cc:407] Connection header received
0925 21:58:35.596514 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:35.596516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:35.596528 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:35.596579 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:35.597500 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.597990 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.598959 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.599225 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.599284 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.601562 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:35.601562 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:35.601563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:35.601569 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:35.603015 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:35.603492 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:35.603492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:35.603493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:35.603554 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:35.603781 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:35.603782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:35.603782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:35.603909 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:35.603971 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:35.604218 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 21:58:35.604288 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 21:58:35.604607 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.595665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:45239)
0925 21:58:35.595848 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:35.595893 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:35.596416 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:35.596649 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:35.596653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:35.597206 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:35.597407 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.597411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.598070 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.598071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.598876 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.598878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.599298 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.599300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.599429 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.599911 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:35.599918 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:35.601464 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:35.603093 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:35.603100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:35.603102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:35.603422 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:35.603605 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:35.603605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:35.603606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:35.603731 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:35.603966 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:35.603970 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:35.604096 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 21:58:35.604185 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 21:58:35.604345 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260925 21:58:35.605034 17054 heartbeater.cc:347] Connected to a master server at 127.2.70.254:46519
I20260925 21:58:35.605120 17054 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:35.605284 17054 heartbeater.cc:511] Master 127.2.70.254:46519 requested a full tablet report, sending...
I20260925 21:58:35.605665 16596 ts_manager.cc:194] Registered new tserver with Master: 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705)
I20260925 21:58:35.606069 16596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:45239
I20260925 21:58:35.607792 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.601257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.2.70.254:46519)
0925 21:58:35.601342 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:35.601343 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:35.601353 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:58:35.601434 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:35.601434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:35.601443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:35.601496 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:35.602107 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.602828 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.603861 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.604148 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.604206 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.605491 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:35.605492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:35.605492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:35.605499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:35.606837 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:35.607136 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:35.607137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:35.607137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:35.607186 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:35.607346 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:35.607346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:35.607347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:35.607443 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:35.607594 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:35.607630 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 21:58:35.607704 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66,"thread_start_us":43,"threads_started":1}
I20260925 21:58:35.608569 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:58:35.610962 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.608864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.2.70.254:46519)
0925 21:58:35.608953 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:35.608954 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:35.608978 (+ 24us) server_negotiation.cc:407] Connection header received
0925 21:58:35.609023 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:35.609024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:35.609053 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:35.609099 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:58:35.609177 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.609626 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.610331 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.610522 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.610573 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.610694 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:58:35.610802 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:58:35.610882 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:35.610897 (+ 15us) server_negotiation.cc:299] Negotiation successful
0925 21:58:35.610916 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 21:58:35.611210 16596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:58:35.612831 16596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:58:35.614763 16718 tablet_service.cc:1467] Processing CreateTablet for tablet fec4f61a20af4540a4380fbe14cb15b0 (DEFAULT_TABLE table=test-table [id=dc6b49d2813045699321395b6c447154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:58:35.614950 16718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec4f61a20af4540a4380fbe14cb15b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:35.616293 16676 tablet_bootstrap.cc:492] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b: Bootstrap starting.
I20260925 21:58:35.616616 16676 tablet_bootstrap.cc:654] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:35.617177 16676 tablet_bootstrap.cc:492] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b: No bootstrap required, opened a new log
I20260925 21:58:35.617230 16676 ts_tablet_manager.cc:1397] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 21:58:35.617334 16676 raft_consensus.cc:348] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:35.617415 16676 raft_consensus.cc:374] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:35.617465 16676 raft_consensus.cc:729] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48653e5edbba40d8a5c7b4e67163a33b, State: Initialized, Role: FOLLOWER
I20260925 21:58:35.617512 16676 consensus_queue.cc:260] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:35.617609 16676 ts_tablet_manager.cc:1428] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:35.618597 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.614536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:38947 (local address 127.0.0.1:57940)
0925 21:58:35.614582 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:35.614594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:35.614636 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:35.615002 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:35.615003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:35.615005 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:35.615047 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.615048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.616445 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.616446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.617426 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.617429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.618351 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.618352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.618420 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.618424 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:58:35.618479 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 21:58:35.618526 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":1}
I20260925 21:58:35.618713 17060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.614626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:38705 (local address 127.0.0.1:43938)
0925 21:58:35.614766 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:35.614774 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:35.614811 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:35.615140 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:35.615141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:35.615143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:35.615182 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.615184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.616599 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.616600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.617516 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.617519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.618533 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.618533 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.618579 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.618582 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:58:35.618615 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 21:58:35.618645 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260925 21:58:35.618887 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.614833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.2.70.195:38705)
0925 21:58:35.615006 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:35.615010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:35.615019 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:58:35.615050 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:35.615052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:35.615066 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:35.615135 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:35.615247 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.616518 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.617590 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.618424 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.618525 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.618581 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:35.618630 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 21:58:35.618736 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260925 21:58:35.618913 17061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.614656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.2.70.194:38947)
0925 21:58:35.614837 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:35.614842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:35.614851 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:58:35.614887 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:35.614889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:35.614905 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:35.615117 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:35.615192 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.616317 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.617557 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.618281 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.618341 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.618386 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:35.618814 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 21:58:35.618856 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":75,"threads_started":1}
I20260925 21:58:35.619518 16856 tablet_service.cc:1467] Processing CreateTablet for tablet fec4f61a20af4540a4380fbe14cb15b0 (DEFAULT_TABLE table=test-table [id=dc6b49d2813045699321395b6c447154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:58:35.619511 16988 tablet_service.cc:1467] Processing CreateTablet for tablet fec4f61a20af4540a4380fbe14cb15b0 (DEFAULT_TABLE table=test-table [id=dc6b49d2813045699321395b6c447154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:58:35.619792 16988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec4f61a20af4540a4380fbe14cb15b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:35.619796 16856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec4f61a20af4540a4380fbe14cb15b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:35.622383 17065 tablet_bootstrap.cc:492] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675: Bootstrap starting.
I20260925 21:58:35.622388 17064 tablet_bootstrap.cc:492] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f: Bootstrap starting.
I20260925 21:58:35.622903 17065 tablet_bootstrap.cc:654] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:35.622931 17064 tablet_bootstrap.cc:654] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:35.623390 17064 log.cc:824] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:35.623416 17065 log.cc:824] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:35.624138 17065 tablet_bootstrap.cc:492] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675: No bootstrap required, opened a new log
I20260925 21:58:35.624173 17064 tablet_bootstrap.cc:492] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f: No bootstrap required, opened a new log
I20260925 21:58:35.624213 17065 ts_tablet_manager.cc:1397] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:58:35.624219 17064 ts_tablet_manager.cc:1397] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:58:35.626053 17064 raft_consensus.cc:348] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:35.626053 17065 raft_consensus.cc:348] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:35.626235 17065 raft_consensus.cc:374] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:35.626235 17064 raft_consensus.cc:374] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:35.626269 17064 raft_consensus.cc:729] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e95fb1bd9349dcbab485f623d7301f, State: Initialized, Role: FOLLOWER
I20260925 21:58:35.626269 17065 raft_consensus.cc:729] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a228ad870e4dc2b264803b3fbf5675, State: Initialized, Role: FOLLOWER
I20260925 21:58:35.626389 17065 consensus_queue.cc:260] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:35.626389 17064 consensus_queue.cc:260] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:35.626653 17064 ts_tablet_manager.cc:1428] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 21:58:35.626653 17065 ts_tablet_manager.cc:1428] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 21:58:35.626760 17054 heartbeater.cc:503] Master 127.2.70.254:46519 was elected leader, sending a full tablet report...
I20260925 21:58:35.626763 16922 heartbeater.cc:503] Master 127.2.70.254:46519 was elected leader, sending a full tablet report...
W20260925 21:58:35.702387 16923 tablet.cc:2369] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 21:58:35.808638 16785 tablet.cc:2369] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:58:35.837056 16789 raft_consensus.cc:482] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:58:35.837190 16789 raft_consensus.cc:504] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:35.837527 16789 leader_election.cc:288] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947), 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705)
I20260925 21:58:35.840404 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.837702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:38947 (local address 127.2.70.193:59207)
0925 21:58:35.837826 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:35.837839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:35.837990 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:35.838151 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:35.838152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:35.838154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:35.838211 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.838213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.838815 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.838817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.839582 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.839584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.840161 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.840162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.840221 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.840225 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:58:35.840269 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 21:58:35.840307 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260925 21:58:35.840476 17061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.837798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:59207 (local address 127.2.70.194:38947)
0925 21:58:35.837855 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:35.837857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:35.837907 (+ 50us) server_negotiation.cc:407] Connection header received
0925 21:58:35.838068 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:35.838069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:35.838083 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:35.838142 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:35.838267 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.838746 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.839668 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.840091 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.840156 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.840164 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:35.840364 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 21:58:35.840406 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 21:58:35.840715 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.837996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:58963 (local address 127.2.70.195:38705)
0925 21:58:35.838067 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:35.838069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:35.838077 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:58:35.838103 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:35.838104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:35.838111 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:35.838153 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:35.838265 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.838733 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.839668 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.840115 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.840298 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.840311 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:35.840600 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 21:58:35.840648 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 21:58:35.840813 16876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec4f61a20af4540a4380fbe14cb15b0" candidate_uuid: "48653e5edbba40d8a5c7b4e67163a33b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84a228ad870e4dc2b264803b3fbf5675" is_pre_election: true
I20260925 21:58:35.840976 16876 raft_consensus.cc:2454] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48653e5edbba40d8a5c7b4e67163a33b in term 0.
I20260925 21:58:35.841027 17071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.837881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:38705 (local address 127.2.70.193:58963)
0925 21:58:35.837968 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:35.837978 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:35.838023 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:35.838163 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:35.838163 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:35.838165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:35.838212 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.838214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.838801 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.838802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.839585 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:35.839588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.840298 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.840299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:35.840372 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.840376 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:58:35.840901 (+ 525us) client_negotiation.cc:240] Negotiation successful
0925 21:58:35.840951 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":4,"thread_start_us":52,"threads_started":1}
I20260925 21:58:35.841213 16671 leader_election.cc:302] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48653e5edbba40d8a5c7b4e67163a33b, 84a228ad870e4dc2b264803b3fbf5675; no voters:
I20260925 21:58:35.841337 16789 raft_consensus.cc:2781] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:58:35.841391 16789 raft_consensus.cc:482] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:58:35.841408 16789 raft_consensus.cc:3037] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:35.841349 17008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec4f61a20af4540a4380fbe14cb15b0" candidate_uuid: "48653e5edbba40d8a5c7b4e67163a33b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e95fb1bd9349dcbab485f623d7301f" is_pre_election: true
I20260925 21:58:35.841496 17008 raft_consensus.cc:2454] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48653e5edbba40d8a5c7b4e67163a33b in term 0.
I20260925 21:58:35.842105 16789 raft_consensus.cc:504] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:35.842217 16789 leader_election.cc:288] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 1 election: Requested vote from peers 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947), 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705)
I20260925 21:58:35.842460 16876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec4f61a20af4540a4380fbe14cb15b0" candidate_uuid: "48653e5edbba40d8a5c7b4e67163a33b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84a228ad870e4dc2b264803b3fbf5675"
I20260925 21:58:35.842473 17008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec4f61a20af4540a4380fbe14cb15b0" candidate_uuid: "48653e5edbba40d8a5c7b4e67163a33b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e95fb1bd9349dcbab485f623d7301f"
I20260925 21:58:35.842540 17008 raft_consensus.cc:3037] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:35.842540 16876 raft_consensus.cc:3037] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:35.843390 16876 raft_consensus.cc:2454] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48653e5edbba40d8a5c7b4e67163a33b in term 1.
I20260925 21:58:35.843390 17008 raft_consensus.cc:2454] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48653e5edbba40d8a5c7b4e67163a33b in term 1.
I20260925 21:58:35.843551 16671 leader_election.cc:302] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48653e5edbba40d8a5c7b4e67163a33b, 84a228ad870e4dc2b264803b3fbf5675; no voters:
I20260925 21:58:35.843637 16789 raft_consensus.cc:2781] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:58:35.843734 16789 raft_consensus.cc:686] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 LEADER]: Becoming Leader. State: Replica: 48653e5edbba40d8a5c7b4e67163a33b, State: Running, Role: LEADER
I20260925 21:58:35.843786 16789 consensus_queue.cc:237] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:35.844487 16596 catalog_manager.cc:5495] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b reported cstate change: term changed from 0 to 1, leader changed from <none> to 48653e5edbba40d8a5c7b4e67163a33b (127.2.70.193). New cstate: current_term: 1 leader_uuid: "48653e5edbba40d8a5c7b4e67163a33b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } health_report { overall_health: HEALTHY } } }
W20260925 21:58:35.845532 17055 tablet.cc:2369] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:58:35.902102 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.899205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.2.70.193:39871)
0925 21:58:35.899353 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:35.899355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:35.899362 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:58:35.899400 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:35.899401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:35.899420 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:35.899489 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:58:35.899626 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.900158 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.901013 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:35.901348 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:35.901433 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:35.901618 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:58:35.901782 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:58:35.901851 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:35.901979 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 21:58:35.902034 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260925 21:58:35.903963 17008 raft_consensus.cc:1261] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f [term 1 FOLLOWER]: Refusing update from remote peer 48653e5edbba40d8a5c7b4e67163a33b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:58:35.903967 16876 raft_consensus.cc:1261] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675 [term 1 FOLLOWER]: Refusing update from remote peer 48653e5edbba40d8a5c7b4e67163a33b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:58:35.904327 16789 consensus_queue.cc:1035] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:58:35.904491 17072 consensus_queue.cc:1035] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:58:35.907241 17078 mvcc.cc:205] Tried to move back new op lower bound from 7333369921139916800 to 7333369920896856064. Current Snapshot: MvccSnapshot[applied={T|T < 7333369921139916800}]
I20260925 21:58:35.908748 16596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:58:35.908828 16596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:58:35.909493 16596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Sending DeleteTablet for 3 replicas of tablet fec4f61a20af4540a4380fbe14cb15b0
I20260925 21:58:35.909766 16718 tablet_service.cc:1514] Processing DeleteTablet for tablet fec4f61a20af4540a4380fbe14cb15b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:35 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:58046
I20260925 21:58:35.909837 16856 tablet_service.cc:1514] Processing DeleteTablet for tablet fec4f61a20af4540a4380fbe14cb15b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:35 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:57940
I20260925 21:58:35.909895 16988 tablet_service.cc:1514] Processing DeleteTablet for tablet fec4f61a20af4540a4380fbe14cb15b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:35 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:43938
I20260925 21:58:35.910116 17087 tablet_replica.cc:330] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675: stopping tablet replica
I20260925 21:58:35.910116 17088 tablet_replica.cc:330] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f: stopping tablet replica
I20260925 21:58:35.910212 17088 raft_consensus.cc:2229] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:58:35.910225 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 16634
I20260925 21:58:35.910322 17088 raft_consensus.cc:2258] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:58:35.910234 17087 raft_consensus.cc:2229] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:58:35.910985 17087 raft_consensus.cc:2258] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:58:35.911029 17088 ts_tablet_manager.cc:1905] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:58:35.911417 17087 ts_tablet_manager.cc:1905] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:58:35.912842 17088 ts_tablet_manager.cc:1918] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:58:35.912914 17088 log.cc:1196] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/fec4f61a20af4540a4380fbe14cb15b0
I20260925 21:58:35.913100 17087 ts_tablet_manager.cc:1918] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:58:35.913156 17088 ts_tablet_manager.cc:1939] T fec4f61a20af4540a4380fbe14cb15b0 P 09e95fb1bd9349dcbab485f623d7301f: Deleting consensus metadata
I20260925 21:58:35.913223 17087 log.cc:1196] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/fec4f61a20af4540a4380fbe14cb15b0
I20260925 21:58:35.913515 17087 ts_tablet_manager.cc:1939] T fec4f61a20af4540a4380fbe14cb15b0 P 84a228ad870e4dc2b264803b3fbf5675: Deleting consensus metadata
I20260925 21:58:35.913539 16581 catalog_manager.cc:4841] TS 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705): tablet fec4f61a20af4540a4380fbe14cb15b0 (table test-table [id=dc6b49d2813045699321395b6c447154]) successfully deleted
I20260925 21:58:35.913936 16581 catalog_manager.cc:4841] TS 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947): tablet fec4f61a20af4540a4380fbe14cb15b0 (table test-table [id=dc6b49d2813045699321395b6c447154]) successfully deleted
W20260925 21:58:35.916185 16583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:39871 (error 108)
I20260925 21:58:35.916436 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 16790
I20260925 21:58:35.917013 17060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:35.916548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:39871 (local address 127.0.0.1:58074)
0925 21:58:35.916940 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:35.916978 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:39871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369}
I20260925 21:58:35.920743 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 16925
W20260925 21:58:35.920636 16583 catalog_manager.cc:4584] TS 48653e5edbba40d8a5c7b4e67163a33b (127.2.70.193:39871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fec4f61a20af4540a4380fbe14cb15b0: Network error: Client connection negotiation failed: client connection to 127.2.70.193:39871: connect: Connection refused (error 111)
I20260925 21:58:35.924458 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 16563
I20260925 21:58:35.928745 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:46519
--webserver_interface=127.2.70.254
--webserver_port=33963
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:46519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:36.008440 17090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:36.008664 17090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:36.008697 17090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:36.009891 17090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:58:36.009931 17090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:36.009950 17090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:36.009966 17090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:58:36.009981 17090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:58:36.011478 17090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:46519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:46519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:36.011730 17090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:36.011957 17090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:36.016655 17098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:36.016685 17096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:36.016741 17090 server_base.cc:1034] running on GCE node
W20260925 21:58:36.016697 17095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:36.017036 17090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:36.017323 17090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:36.018471 17090 hybrid_clock.cc:648] HybridClock initialized: now 1790373516018456 us; error 26 us; skew 500 ppm
Sep 25 21:58:36 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:36.020952 17090 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:58:36.021368 17090 webserver.cc:460] Webserver started at http://127.2.70.254:33963/ using document root <none> and password file <none>
I20260925 21:58:36.021584 17090 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:36.021634 17090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:36.022994 17090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:36.023602 17105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:36.023761 17090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:58:36.023865 17090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "9073fed4836b48b9b0bf8154a09dbb49"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "0d9b68d9efcc3ec5207156474573e55f"
server_key_iv: "4033802b6a467c2826032e3b30ddc7ea"
server_key_version: "encryptionkey@0"
I20260925 21:58:36.024214 17090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:36.045656 17090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:36.047780 17090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:36.047945 17090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:36.051716 17090 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:46519
I20260925 21:58:36.051750 17157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:46519 every 8 connection(s)
I20260925 21:58:36.052140 17090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:58:36.053241 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 17090
I20260925 21:58:36.053498 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:39871
--local_ip_for_outbound_sockets=127.2.70.193
--tserver_master_addrs=127.2.70.254:46519
--webserver_port=36669
--webserver_interface=127.2.70.193
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:58:36.053783 17158 sys_catalog.cc:263] Verifying existing consensus state
I20260925 21:58:36.054471 17158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Bootstrap starting.
I20260925 21:58:36.063109 17158 log.cc:824] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:36.065856 17158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 21:58:36.066193 17158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49: Bootstrap complete.
I20260925 21:58:36.068501 17158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } }
I20260925 21:58:36.068758 17158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9073fed4836b48b9b0bf8154a09dbb49, State: Initialized, Role: FOLLOWER
I20260925 21:58:36.068933 17158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } }
I20260925 21:58:36.069005 17158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:58:36.069041 17158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:58:36.069092 17158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 2 FOLLOWER]: Advancing to term 3
I20260925 21:58:36.070165 17158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } }
I20260925 21:58:36.070278 17158 leader_election.cc:302] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9073fed4836b48b9b0bf8154a09dbb49; no voters:
I20260925 21:58:36.070478 17158 leader_election.cc:288] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 21:58:36.070533 17162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 21:58:36.070673 17162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [term 3 LEADER]: Becoming Leader. State: Replica: 9073fed4836b48b9b0bf8154a09dbb49, State: Running, Role: LEADER
I20260925 21:58:36.070782 17158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:58:36.070786 17162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } }
I20260925 21:58:36.071110 17162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9073fed4836b48b9b0bf8154a09dbb49. Latest consensus state: current_term: 3 leader_uuid: "9073fed4836b48b9b0bf8154a09dbb49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } } }
I20260925 21:58:36.071189 17162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:36.071355 17164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9073fed4836b48b9b0bf8154a09dbb49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073fed4836b48b9b0bf8154a09dbb49" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46519 } } }
I20260925 21:58:36.071451 17164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9073fed4836b48b9b0bf8154a09dbb49 [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:36.071544 17166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:58:36.072394 17166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:58:36.072507 17166 catalog_manager.cc:1259] Loaded cluster ID: 5d2d9093cb9b4e038f6e5d1916867da5
I20260925 21:58:36.072538 17166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 21:58:36.072737 17166 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6d5bfcf9ad google::LogMessage::Fail() at ??:0
@ 0x7f6d5bfd288a google::LogMessage::SendToLog() at ??:0
@ 0x7f6d5bfcf4de google::LogMessage::Flush() at ??:0
@ 0x7f6d5bfd0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6d63ef40f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6d63efd8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6d5de9d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6d5de92954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6d5ecf66db start_thread at ??:0
@ 0x7f6d5aacd61f clone at ??:0
W20260925 21:58:36.148607 17160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:36.148813 17160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:36.148840 17160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:36.150125 17160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:36.150172 17160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:36.150200 17160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:58:36.151726 17160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:39871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=36669
--tserver_master_addrs=127.2.70.254:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:36.151984 17160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:36.152212 17160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:36.157202 17197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:36.157223 17195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:36.157315 17194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:36.157538 17160 server_base.cc:1034] running on GCE node
I20260925 21:58:36.157763 17160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:36.158062 17160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:36.159251 17160 hybrid_clock.cc:648] HybridClock initialized: now 1790373516159231 us; error 34 us; skew 500 ppm
Sep 25 21:58:36 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:36.162468 17160 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:58:36.163043 17160 webserver.cc:460] Webserver started at http://127.2.70.193:36669/ using document root <none> and password file <none>
I20260925 21:58:36.163312 17160 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:36.163379 17160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:36.165246 17160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:36.166106 17204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:36.166297 17160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:36.166427 17160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "48653e5edbba40d8a5c7b4e67163a33b"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "26469f8aa2fa5bef7ff8b7e7b647220e"
server_key_iv: "9f924744dcd1b6d00bd4548b79c17f27"
server_key_version: "encryptionkey@0"
I20260925 21:58:36.166909 17160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:36.194659 17160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:36.197355 17160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:36.197532 17160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:36.197808 17160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:36.198297 17211 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 21:58:36.199787 17160 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 21:58:36.199864 17160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 21:58:36.199900 17160 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 21:58:36.200826 17160 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 21:58:36.200874 17160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260925 21:58:36.200876 17211 tablet_bootstrap.cc:492] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b: Bootstrap starting.
I20260925 21:58:36.207921 17160 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:39871
I20260925 21:58:36.208127 17318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:39871 every 8 connection(s)
I20260925 21:58:36.208395 17160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:58:36.209192 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 17160
I20260925 21:58:36.209435 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:38947
--local_ip_for_outbound_sockets=127.2.70.194
--tserver_master_addrs=127.2.70.254:46519
--webserver_port=42351
--webserver_interface=127.2.70.194
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:58:36.211205 17211 log.cc:824] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:36.213845 17211 tablet_bootstrap.cc:492] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 21:58:36.214145 17211 tablet_bootstrap.cc:492] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b: Bootstrap complete.
I20260925 21:58:36.215263 17211 ts_tablet_manager.cc:1397] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260925 21:58:36.216714 17211 raft_consensus.cc:348] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:36.217049 17211 raft_consensus.cc:729] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48653e5edbba40d8a5c7b4e67163a33b, State: Initialized, Role: FOLLOWER
I20260925 21:58:36.217213 17211 consensus_queue.cc:260] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:36.217492 17211 ts_tablet_manager.cc:1428] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
W20260925 21:58:36.278751 17321 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:58:36.209081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:49815)
0925 21:58:36.209309 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:36.209358 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:36.210276 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:36.278560 (+ 68284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
W20260925 21:58:36.279049 17319 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:46519 (0 consecutive failures): Network error: Failed to ping master at 127.2.70.254:46519: Client connection negotiation failed: client connection to 127.2.70.254:46519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 21:58:36.279394 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:36.279271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:56281)
0925 21:58:36.279322 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:36.279373 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 21:58:36.279770 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:36.279653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:42347)
0925 21:58:36.279717 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:36.279747 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 21:58:36.279917 17319 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 21:58:36.292735 17322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:36.292948 17322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:36.292973 17322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:36.294153 17322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:36.294194 17322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:36.294220 17322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:58:36.295696 17322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:38947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=42351
--tserver_master_addrs=127.2.70.254:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:36.295948 17322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:36.296166 17322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:36.301229 17332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:36.301277 17329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:36.301277 17330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:36.301440 17322 server_base.cc:1034] running on GCE node
I20260925 21:58:36.301662 17322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:36.301883 17322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:36.303033 17322 hybrid_clock.cc:648] HybridClock initialized: now 1790373516303015 us; error 32 us; skew 500 ppm
Sep 25 21:58:36 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:36.305615 17322 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:58:36.306041 17322 webserver.cc:460] Webserver started at http://127.2.70.194:42351/ using document root <none> and password file <none>
I20260925 21:58:36.306238 17322 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:36.306279 17322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:36.307617 17322 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:36.308337 17339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:36.308491 17322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:36.308578 17322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "84a228ad870e4dc2b264803b3fbf5675"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "66d1cac6897afbffd987352cb8ac02e5"
server_key_iv: "e5b1ed405ab7f7bbbe447c165f177669"
server_key_version: "encryptionkey@0"
I20260925 21:58:36.308946 17322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:36.318776 17322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:36.320892 17322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:36.321017 17322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:36.321254 17322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:36.321559 17322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:36.321592 17322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:36.321614 17322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:36.321636 17322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:36.327255 17322 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:38947
I20260925 21:58:36.327312 17452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:38947 every 8 connection(s)
I20260925 21:58:36.327625 17322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:58:36.328646 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:36.328271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:45293)
0925 21:58:36.328422 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:36.328494 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
W20260925 21:58:36.329007 17453 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:46519 (0 consecutive failures): Network error: Failed to ping master at 127.2.70.254:46519: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
I20260925 21:58:36.329308 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:36.329225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:38109)
0925 21:58:36.329264 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:36.329290 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 21:58:36.329523 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:36.329465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:39379)
0925 21:58:36.329494 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:36.329511 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 21:58:36.329630 17453 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 21:58:36.335628 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 17322
I20260925 21:58:36.335920 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:38705
--local_ip_for_outbound_sockets=127.2.70.195
--tserver_master_addrs=127.2.70.254:46519
--webserver_port=44889
--webserver_interface=127.2.70.195
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:36.416919 17456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:36.417152 17456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:36.417178 17456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:36.418432 17456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:36.418480 17456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:36.418509 17456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:58:36.420083 17456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:38705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=44889
--tserver_master_addrs=127.2.70.254:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:36.420321 17456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:36.420550 17456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:36.425807 17462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:36.425817 17456 server_base.cc:1034] running on GCE node
W20260925 21:58:36.425938 17464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:36.425813 17461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:36.426400 17456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:36.426628 17456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:36.427798 17456 hybrid_clock.cc:648] HybridClock initialized: now 1790373516427781 us; error 28 us; skew 500 ppm
Sep 25 21:58:36 dist-test-slave-2g0f krb5kdc[16017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:36.430461 17456 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:58:36.430886 17456 webserver.cc:460] Webserver started at http://127.2.70.195:44889/ using document root <none> and password file <none>
I20260925 21:58:36.431095 17456 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:36.431144 17456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:36.432605 17456 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:36.433259 17471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:36.433437 17456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:58:36.433527 17456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "09e95fb1bd9349dcbab485f623d7301f"
format_stamp: "Formatted at 2026-09-25 21:58:34 on dist-test-slave-2g0f"
server_key: "e69630fa80086d289d3857098466e4a8"
server_key_iv: "031aba9fd9b7d885fd3ebc284b9594f3"
server_key_version: "encryptionkey@0"
I20260925 21:58:36.433964 17456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:36.449371 17456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:36.451655 17456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:36.451776 17456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:36.452052 17456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:36.452332 17456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:36.452361 17456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:36.452389 17456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:36.452401 17456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:36.457901 17456 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:38705
I20260925 21:58:36.457942 17584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:38705 every 8 connection(s)
I20260925 21:58:36.458283 17456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:58:36.459350 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:36.458959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:40827)
0925 21:58:36.459123 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:36.459205 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
W20260925 21:58:36.459728 17585 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:46519 (0 consecutive failures): Network error: Failed to ping master at 127.2.70.254:46519: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
I20260925 21:58:36.460072 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:36.459944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:39527)
0925 21:58:36.460015 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:36.460048 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 21:58:36.460315 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:36.460260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:49903)
0925 21:58:36.460283 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:36.460303 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 21:58:36.460395 17585 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 21:58:36.460922 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 17456
I20260925 21:58:37.280522 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:37.280256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:34111)
0925 21:58:37.280398 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:37.280458 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260925 21:58:37.330163 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:37.329958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:52959)
0925 21:58:37.330070 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:37.330111 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260925 21:58:37.460937 17591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:37.460718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:60735)
0925 21:58:37.460843 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:37.460878 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260925 21:58:37.662241 17592 raft_consensus.cc:482] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:58:37.662369 17592 raft_consensus.cc:504] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:37.662676 17592 leader_election.cc:288] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947), 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705)
Sep 25 21:58:37 dist-test-slave-2g0f krb5kdc[16017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.195@KRBTEST.COM
Sep 25 21:58:37 dist-test-slave-2g0f krb5kdc[16017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.194@KRBTEST.COM
I20260925 21:58:37.674081 17595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:37.663071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:43479 (local address 127.2.70.195:38705)
0925 21:58:37.663214 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:37.663220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:37.663244 (+ 24us) server_negotiation.cc:407] Connection header received
0925 21:58:37.663814 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:37.663817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:37.663897 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:37.664045 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:37.664984 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:37.666417 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:37.667937 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:37.668531 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:37.668641 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:37.670886 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:37.670891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:37.670892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:37.670905 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:37.672571 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:37.672948 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:37.672951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:37.672951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:37.673044 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:37.673257 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:37.673258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:37.673258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:37.673492 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:37.673563 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:37.673831 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 21:58:37.673954 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":50,"threads_started":1}
I20260925 21:58:37.674078 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:37.662958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:38705 (local address 127.2.70.193:43479)
0925 21:58:37.663148 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:37.663157 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:37.663210 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:37.664115 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:37.664118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:37.664590 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:37.664875 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:37.664884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:37.666531 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:37.666532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:37.667822 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:37.667826 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:37.668619 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:37.668620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:37.668670 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:37.669257 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:37.669261 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:37.670760 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:37.672646 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:37.672653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:37.672655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:37.672869 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:37.673102 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:37.673103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:37.673104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:37.673202 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:37.673558 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:37.673565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:37.673688 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 21:58:37.673773 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 21:58:37.673985 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":67,"thread_start_us":126,"threads_started":1}
I20260925 21:58:37.674093 17593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:37.662972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:55681 (local address 127.2.70.194:38947)
0925 21:58:37.663185 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:37.663197 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:37.663233 (+ 36us) server_negotiation.cc:407] Connection header received
0925 21:58:37.663806 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:37.663810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:37.663897 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:37.664045 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:37.664992 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:37.666425 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:37.667933 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:37.668441 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:37.668525 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:37.671227 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:37.671232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:37.671232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:37.671245 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:37.672826 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:37.673134 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:37.673137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:37.673137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:37.673227 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:37.673391 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:37.673392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:37.673392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:37.673617 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:37.673683 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:37.673850 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 21:58:37.673971 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":99,"threads_started":1}
I20260925 21:58:37.674171 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:37.662839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:38947 (local address 127.2.70.193:55681)
0925 21:58:37.662889 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:37.662901 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:37.662958 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:37.664115 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:37.664118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:37.664590 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:37.664875 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:37.664884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:37.666547 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:37.666548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:37.667822 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:37.667853 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:37.668540 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:37.668541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:37.668667 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:37.669216 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:37.669225 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:37.671092 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:37.672902 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:37.672903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:37.672903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:37.673060 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:37.673284 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:37.673285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:37.673285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:37.673342 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:37.673677 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:37.673678 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:37.673721 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 21:58:37.673786 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 21:58:37.674059 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":208}
W20260925 21:58:37.675005 17206 leader_election.cc:341] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947): Not found: Tablet not found: fec4f61a20af4540a4380fbe14cb15b0
W20260925 21:58:37.675530 17206 leader_election.cc:341] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705): Not found: Tablet not found: fec4f61a20af4540a4380fbe14cb15b0
I20260925 21:58:37.675585 17206 leader_election.cc:302] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48653e5edbba40d8a5c7b4e67163a33b; no voters: 09e95fb1bd9349dcbab485f623d7301f, 84a228ad870e4dc2b264803b3fbf5675
I20260925 21:58:37.675685 17592 raft_consensus.cc:2726] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 21:58:38.281241 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:38.281018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:40889)
0925 21:58:38.281152 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:38.281192 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260925 21:58:38.330866 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:38.330645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:37133)
0925 21:58:38.330792 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:38.330821 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260925 21:58:38.461660 17598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:38.461437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:59751)
0925 21:58:38.461576 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:38.461615 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260925 21:58:39.281949 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:39.281715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:59937)
0925 21:58:39.281860 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:39.281901 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260925 21:58:39.331439 17600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:39.331264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:44337)
0925 21:58:39.331366 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:39.331397 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
I20260925 21:58:39.462378 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:39.462132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:44677)
0925 21:58:39.462275 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:39.462321 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260925 21:58:39.905525 17602 raft_consensus.cc:482] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:58:39.905622 17602 raft_consensus.cc:504] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:39.905792 17602 leader_election.cc:288] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947), 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705)
W20260925 21:58:39.906284 17206 leader_election.cc:341] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705): Not found: Tablet not found: fec4f61a20af4540a4380fbe14cb15b0
W20260925 21:58:39.906348 17206 leader_election.cc:341] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947): Not found: Tablet not found: fec4f61a20af4540a4380fbe14cb15b0
I20260925 21:58:39.906369 17206 leader_election.cc:302] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48653e5edbba40d8a5c7b4e67163a33b; no voters: 09e95fb1bd9349dcbab485f623d7301f, 84a228ad870e4dc2b264803b3fbf5675
I20260925 21:58:39.906432 17602 raft_consensus.cc:2726] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 21:58:40.282683 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:40.282449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:53535)
0925 21:58:40.282593 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:40.282634 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260925 21:58:40.332144 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:40.331892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:39915)
0925 21:58:40.332041 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:40.332085 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260925 21:58:40.462993 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:40.462784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:44459)
0925 21:58:40.462912 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:40.462950 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260925 21:58:41.283540 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:41.283270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:54627)
0925 21:58:41.283429 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:41.283469 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260925 21:58:41.332803 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:41.332581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:59167)
0925 21:58:41.332717 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:41.332758 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260925 21:58:41.463743 17608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:41.463478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:56613)
0925 21:58:41.463641 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:41.463686 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260925 21:58:41.859155 17609 raft_consensus.cc:482] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:58:41.859253 17609 raft_consensus.cc:504] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:41.859416 17609 leader_election.cc:288] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947), 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705)
W20260925 21:58:41.859812 17206 leader_election.cc:341] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947): Not found: Tablet not found: fec4f61a20af4540a4380fbe14cb15b0
W20260925 21:58:41.859966 17206 leader_election.cc:341] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705): Not found: Tablet not found: fec4f61a20af4540a4380fbe14cb15b0
I20260925 21:58:41.859993 17206 leader_election.cc:302] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48653e5edbba40d8a5c7b4e67163a33b; no voters: 09e95fb1bd9349dcbab485f623d7301f, 84a228ad870e4dc2b264803b3fbf5675
I20260925 21:58:41.860080 17609 raft_consensus.cc:2726] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 21:58:42.284284 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:42.284074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:59489)
0925 21:58:42.284202 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:42.284238 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260925 21:58:42.333634 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:42.333366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:48023)
0925 21:58:42.333533 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:42.333577 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260925 21:58:42.464378 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:42.464177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:48297)
0925 21:58:42.464298 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:42.464334 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260925 21:58:43.285050 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:43.284753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:46273)
0925 21:58:43.284960 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:43.285002 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":65,"threads_started":1}
I20260925 21:58:43.334342 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:43.334107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:58553)
0925 21:58:43.334242 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:43.334285 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260925 21:58:43.362370 17615 raft_consensus.cc:482] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:58:43.362462 17615 raft_consensus.cc:504] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:43.362640 17615 leader_election.cc:288] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947), 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705)
W20260925 21:58:43.363036 17206 leader_election.cc:341] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947): Not found: Tablet not found: fec4f61a20af4540a4380fbe14cb15b0
W20260925 21:58:43.363101 17206 leader_election.cc:341] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705): Not found: Tablet not found: fec4f61a20af4540a4380fbe14cb15b0
I20260925 21:58:43.363116 17206 leader_election.cc:302] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48653e5edbba40d8a5c7b4e67163a33b; no voters: 09e95fb1bd9349dcbab485f623d7301f, 84a228ad870e4dc2b264803b3fbf5675
I20260925 21:58:43.363204 17615 raft_consensus.cc:2726] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 21:58:43.465070 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:43.464810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:58659)
0925 21:58:43.464943 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:43.465024 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260925 21:58:44.285851 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:44.285591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:55895)
0925 21:58:44.285759 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:44.285804 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
I20260925 21:58:44.335038 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:44.334794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:38889)
0925 21:58:44.334951 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:44.334991 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":60,"threads_started":1}
I20260925 21:58:44.465852 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:44.465606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:38103)
0925 21:58:44.465752 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:44.465805 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260925 21:58:45.286636 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:45.286410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:36401)
0925 21:58:45.286543 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:45.286588 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260925 21:58:45.335706 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:45.335508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:40787)
0925 21:58:45.335622 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:45.335659 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260925 21:58:45.466603 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:45.466375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:58777)
0925 21:58:45.466502 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:45.466552 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260925 21:58:46.287446 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:46.287178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:40483)
0925 21:58:46.287344 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:46.287399 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260925 21:58:46.336339 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:46.336114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:53121)
0925 21:58:46.336254 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:46.336288 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260925 21:58:46.467365 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:46.467130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:38703)
0925 21:58:46.467276 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:46.467317 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260925 21:58:47.288337 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:47.288066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:36209)
0925 21:58:47.288232 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:47.288286 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260925 21:58:47.337093 17627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:47.336833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:44081)
0925 21:58:47.337004 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:47.337039 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":117,"threads_started":1}
I20260925 21:58:47.468129 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:47.467913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:54531)
0925 21:58:47.468034 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:47.468079 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260925 21:58:48.289240 17629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:48.288869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:52983)
0925 21:58:48.289132 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:48.289179 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":134,"threads_started":1}
I20260925 21:58:48.337740 17630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:48.337517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:59497)
0925 21:58:48.337647 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:48.337690 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260925 21:58:48.468822 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:48.468578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:46859)
0925 21:58:48.468728 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:48.468772 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260925 21:58:49.205004 17632 raft_consensus.cc:482] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:58:49.205130 17632 raft_consensus.cc:504] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a228ad870e4dc2b264803b3fbf5675" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 38947 } } peers { permanent_uuid: "09e95fb1bd9349dcbab485f623d7301f" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 38705 } } peers { permanent_uuid: "48653e5edbba40d8a5c7b4e67163a33b" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 39871 } }
I20260925 21:58:49.205279 17632 leader_election.cc:288] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947), 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705)
W20260925 21:58:49.205713 17206 leader_election.cc:341] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09e95fb1bd9349dcbab485f623d7301f (127.2.70.195:38705): Not found: Tablet not found: fec4f61a20af4540a4380fbe14cb15b0
W20260925 21:58:49.205778 17206 leader_election.cc:341] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84a228ad870e4dc2b264803b3fbf5675 (127.2.70.194:38947): Not found: Tablet not found: fec4f61a20af4540a4380fbe14cb15b0
I20260925 21:58:49.205793 17206 leader_election.cc:302] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48653e5edbba40d8a5c7b4e67163a33b; no voters: 09e95fb1bd9349dcbab485f623d7301f, 84a228ad870e4dc2b264803b3fbf5675
I20260925 21:58:49.205860 17632 raft_consensus.cc:2726] T fec4f61a20af4540a4380fbe14cb15b0 P 48653e5edbba40d8a5c7b4e67163a33b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 21:58:49.290091 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:49.289805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:50253)
0925 21:58:49.289946 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:49.289993 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260925 21:58:49.338430 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:49.338182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:41159)
0925 21:58:49.338342 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:49.338382 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260925 21:58:49.469626 17635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:49.469398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:57201)
0925 21:58:49.469513 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:49.469555 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260925 21:58:50.290858 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:50.290616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:42623)
0925 21:58:50.290768 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:50.290811 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260925 21:58:50.339116 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:50.338878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:36333)
0925 21:58:50.339028 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:50.339062 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260925 21:58:50.470369 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:50.470153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:49287)
0925 21:58:50.470279 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:50.470322 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260925 21:58:51.291791 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:51.291509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.193:49257)
0925 21:58:51.291688 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:51.291742 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260925 21:58:51.339893 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:51.339659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.194:59787)
0925 21:58:51.339775 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:51.339811 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260925 21:58:51.464093 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 17160
I20260925 21:58:51.468741 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 17322
I20260925 21:58:51.471086 17641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:51.470861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:46519 (local address 127.2.70.195:59599)
0925 21:58:51.471004 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:51.471041 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":57,"threads_started":1}
I20260925 21:58:51.473445 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 17456
2026-09-25T21:58:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18386 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:58:51 dist-test-slave-2g0f krb5kdc[17644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:58:51 dist-test-slave-2g0f krb5kdc[17644](info): set up 2 sockets
Sep 25 21:58:51 dist-test-slave-2g0f krb5kdc[17644](info): commencing operation
krb5kdc: starting...
W20260925 21:58:52.439731 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:58:52 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:58:52.557034 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:32829
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:52.638888 17657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:52.639174 17657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:52.639210 17657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:52.640941 17657 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 21:58:52.641010 17657 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 21:58:52.641031 17657 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 21:58:52.641047 17657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:58:52.641074 17657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:52.641094 17657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:52.641119 17657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:58:52.641144 17657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:58:52.643338 17657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:32829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:52.643636 17657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:52.643908 17657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:58:52.649210 17657 server_base.cc:1034] running on GCE node
W20260925 21:58:52.649405 17665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:52.649320 17662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:52.649320 17663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:52.649760 17657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:52.650058 17657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:52.651211 17657 hybrid_clock.cc:648] HybridClock initialized: now 1790373532651180 us; error 42 us; skew 500 ppm
Sep 25 21:58:52 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373532, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:52.653553 17657 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:58:52.653970 17657 webserver.cc:460] Webserver started at http://127.2.70.254:44703/ using document root <none> and password file <none>
I20260925 21:58:52.654152 17657 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:52.654191 17657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:52.654276 17657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:58:52.655279 17657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "5ec1ce98a8614f5d8424eaea5faaa95d"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "ea4232ad2fd7daeebcd782d96ed02e8f"
server_key_iv: "56768b35d0bdd539a83f7eafdc52d848"
server_key_version: "encryptionkey@0"
I20260925 21:58:52.655565 17657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "5ec1ce98a8614f5d8424eaea5faaa95d"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "ea4232ad2fd7daeebcd782d96ed02e8f"
server_key_iv: "56768b35d0bdd539a83f7eafdc52d848"
server_key_version: "encryptionkey@0"
I20260925 21:58:52.656783 17657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:58:52.657492 17672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:52.657672 17657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:58:52.657763 17657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "5ec1ce98a8614f5d8424eaea5faaa95d"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "ea4232ad2fd7daeebcd782d96ed02e8f"
server_key_iv: "56768b35d0bdd539a83f7eafdc52d848"
server_key_version: "encryptionkey@0"
I20260925 21:58:52.657819 17657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:52.668198 17657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:52.670259 17657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:52.670389 17657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:52.674021 17657 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:32829
I20260925 21:58:52.674081 17724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:32829 every 8 connection(s)
I20260925 21:58:52.674393 17657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:58:52.675251 17725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:52.677538 17725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Bootstrap starting.
I20260925 21:58:52.678087 17725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:52.678355 17725 log.cc:824] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:52.678978 17725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: No bootstrap required, opened a new log
I20260925 21:58:52.680656 17725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } }
I20260925 21:58:52.680796 17725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:52.680827 17725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec1ce98a8614f5d8424eaea5faaa95d, State: Initialized, Role: FOLLOWER
I20260925 21:58:52.680955 17725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } }
I20260925 21:58:52.681015 17725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:58:52.681046 17725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:58:52.681092 17725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:52.681607 17725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } }
I20260925 21:58:52.681699 17725 leader_election.cc:302] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ec1ce98a8614f5d8424eaea5faaa95d; no voters:
I20260925 21:58:52.681860 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 17657
I20260925 21:58:52.681867 17725 leader_election.cc:288] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:58:52.681907 17728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:58:52.681950 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:58:52.682013 17728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 1 LEADER]: Becoming Leader. State: Replica: 5ec1ce98a8614f5d8424eaea5faaa95d, State: Running, Role: LEADER
I20260925 21:58:52.682130 17725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:58:52.682200 2331 external_mini_cluster.cc:1428] Setting key c068188705fdf0c496fda8f344fa04a5
I20260925 21:58:52.682160 17728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } }
I20260925 21:58:52.682804 17730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ec1ce98a8614f5d8424eaea5faaa95d. Latest consensus state: current_term: 1 leader_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } } }
I20260925 21:58:52.682822 17729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } } }
I20260925 21:58:52.682936 17730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:52.682950 17729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: This master's current role is: LEADER
I20260925 21:58:52.683156 17739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:58:52.683750 17739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:58:52.686597 17739 catalog_manager.cc:1347] Generated new cluster ID: 747b2058585c4b55baba0ff84a02e69a
I20260925 21:58:52.686658 17739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 21:58:52 dist-test-slave-2g0f krb5kdc[17644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:52.692982 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:52.682721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.2.70.254:32829)
0925 21:58:52.683795 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:52.683805 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:52.683866 (+ 61us) server_negotiation.cc:407] Connection header received
0925 21:58:52.683995 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:52.684012 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:52.684132 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:52.684359 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:52.685061 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:52.686447 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:52.687251 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:52.687675 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:52.687768 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:52.689953 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:52.689957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:52.689957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:52.689967 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:52.691485 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:52.691859 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:52.691862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:52.691862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:52.691954 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:52.692132 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:52.692133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:52.692134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:52.692393 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:52.692545 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:52.692618 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 21:58:52.692759 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1028,"thread_start_us":81,"threads_started":1}
I20260925 21:58:52.713599 17739 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:58:52.714258 17739 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:58:52.717877 17739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Generated new TSK 0
I20260925 21:58:52.718093 17739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:58:52.747455 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:32829
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:52.827767 17751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:52.827989 17751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:52.828011 17751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:52.829170 17751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:52.829210 17751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:52.829236 17751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:58:52.830709 17751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:52.830935 17751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:52.831140 17751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:52.835964 17759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:52.835991 17757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:52.835968 17756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:52.836210 17751 server_base.cc:1034] running on GCE node
I20260925 21:58:52.836474 17751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:52.836678 17751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:52.837824 17751 hybrid_clock.cc:648] HybridClock initialized: now 1790373532837805 us; error 37 us; skew 500 ppm
Sep 25 21:58:52 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373532, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:52.840575 17751 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:58:52.841099 17751 webserver.cc:460] Webserver started at http://127.2.70.193:36157/ using document root <none> and password file <none>
I20260925 21:58:52.841372 17751 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:52.841445 17751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:52.841590 17751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:58:52.842965 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "6e43ab7079304b4999f0f6927d2b3c0d"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "24f3f27d88442f2450199f598dd02ad6"
server_key_iv: "60e0d8153764eda7e9cdf6b5b56fd7a2"
server_key_version: "encryptionkey@0"
I20260925 21:58:52.843341 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "6e43ab7079304b4999f0f6927d2b3c0d"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "24f3f27d88442f2450199f598dd02ad6"
server_key_iv: "60e0d8153764eda7e9cdf6b5b56fd7a2"
server_key_version: "encryptionkey@0"
I20260925 21:58:52.844638 17751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:58:52.845369 17766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:52.845527 17751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:58:52.845602 17751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "6e43ab7079304b4999f0f6927d2b3c0d"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "24f3f27d88442f2450199f598dd02ad6"
server_key_iv: "60e0d8153764eda7e9cdf6b5b56fd7a2"
server_key_version: "encryptionkey@0"
I20260925 21:58:52.845669 17751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:52.853174 17751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:52.854928 17751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:52.855067 17751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:52.855271 17751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:52.855558 17751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:52.855589 17751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:52.855618 17751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:52.855641 17751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:52.861130 17751 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:33061
I20260925 21:58:52.861188 17879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:33061 every 8 connection(s)
I20260925 21:58:52.861519 17751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:58:52.862035 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 17751
I20260925 21:58:52.862143 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:58:52.862361 2331 external_mini_cluster.cc:1428] Setting key 0ed9d857a26e050e7a33b573a7fa00fc
Sep 25 21:58:52 dist-test-slave-2g0f krb5kdc[17644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373532, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
I20260925 21:58:52.871088 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:52.862521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:44137 (local address 127.2.70.254:32829)
0925 21:58:52.862570 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:52.862572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:52.862583 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:58:52.863074 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:52.863075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:52.863086 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:52.863153 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:52.864203 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:52.864757 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:52.865773 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:52.865985 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:52.866041 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:52.868363 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:52.868364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:52.868365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:52.868373 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:52.869759 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:52.870157 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:52.870159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:52.870160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:52.870216 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:52.870431 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:52.870432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:52.870432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:52.870549 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:52.870618 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:52.870898 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 21:58:52.870996 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 21:58:52.871240 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:52.862218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:44137)
0925 21:58:52.862380 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:52.862438 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:52.862997 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:52.863211 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:52.863215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:52.863782 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:52.864055 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:52.864063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:52.864846 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:52.864847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:52.865693 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:52.865696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:52.866064 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:52.866065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:52.866208 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:52.866696 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:52.866705 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:52.868262 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:52.869837 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:52.869843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:52.869846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:52.870086 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:52.870272 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:52.870273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:52.870273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:52.870377 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:52.870616 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:52.870622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:52.870766 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 21:58:52.870865 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 21:58:52.871026 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260925 21:58:52.871681 17880 heartbeater.cc:347] Connected to a master server at 127.2.70.254:32829
I20260925 21:58:52.871809 17880 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:52.872023 17880 heartbeater.cc:511] Master 127.2.70.254:32829 requested a full tablet report, sending...
I20260925 21:58:52.872545 17689 ts_manager.cc:194] Registered new tserver with Master: 6e43ab7079304b4999f0f6927d2b3c0d (127.2.70.193:33061)
I20260925 21:58:52.873116 17689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:44137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:58:52.890407 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:32829
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:52.970784 17885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:52.971011 17885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:52.971042 17885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:52.972266 17885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:52.972314 17885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:52.972350 17885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:58:52.973889 17885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:52.974129 17885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:52.974351 17885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:52.979290 17891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:52.979295 17893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:52.979293 17890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:52.979516 17885 server_base.cc:1034] running on GCE node
I20260925 21:58:52.979768 17885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:52.980135 17885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:52.981276 17885 hybrid_clock.cc:648] HybridClock initialized: now 1790373532981266 us; error 27 us; skew 500 ppm
Sep 25 21:58:52 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373532, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:52.983855 17885 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:58:52.984289 17885 webserver.cc:460] Webserver started at http://127.2.70.194:45607/ using document root <none> and password file <none>
I20260925 21:58:52.984500 17885 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:52.984541 17885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:52.984633 17885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:58:52.985672 17885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "1d1d335cbcbc459ca9eab6995926b1d8"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "ebd36810ec28cec268ac367f87f362e4"
server_key_iv: "1f8ef791e75cc9e81df39baa9a8e274d"
server_key_version: "encryptionkey@0"
I20260925 21:58:52.985965 17885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "1d1d335cbcbc459ca9eab6995926b1d8"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "ebd36810ec28cec268ac367f87f362e4"
server_key_iv: "1f8ef791e75cc9e81df39baa9a8e274d"
server_key_version: "encryptionkey@0"
I20260925 21:58:52.987332 17885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:58:52.988191 17900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:52.988322 17885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:58:52.988402 17885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "1d1d335cbcbc459ca9eab6995926b1d8"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "ebd36810ec28cec268ac367f87f362e4"
server_key_iv: "1f8ef791e75cc9e81df39baa9a8e274d"
server_key_version: "encryptionkey@0"
I20260925 21:58:52.988495 17885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:52.996367 17885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:52.998443 17885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:52.998569 17885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:52.998786 17885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:52.999080 17885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:52.999114 17885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:52.999135 17885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:52.999157 17885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:53.004657 17885 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:35691
I20260925 21:58:53.004693 18013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:35691 every 8 connection(s)
I20260925 21:58:53.004985 17885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:58:53.005227 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 17885
I20260925 21:58:53.005338 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:58:53.005544 2331 external_mini_cluster.cc:1428] Setting key c1f9423ac602e4e842861c55add948ce
Sep 25 21:58:53 dist-test-slave-2g0f krb5kdc[17644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373532, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
I20260925 21:58:53.016350 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.005943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:50583 (local address 127.2.70.254:32829)
0925 21:58:53.005990 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:53.005992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:53.006263 (+ 271us) server_negotiation.cc:407] Connection header received
0925 21:58:53.006880 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:53.006882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:53.006891 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:53.006930 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:53.008309 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.008842 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.010316 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.010571 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.010648 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.013283 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:53.013284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:53.013285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:53.013292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:53.014850 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:53.015347 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:53.015348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:53.015349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:53.015407 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:53.015695 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:53.015696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:53.015696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:53.015859 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:53.015920 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:53.016199 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 21:58:53.016279 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 21:58:53.016531 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.005979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:50583)
0925 21:58:53.006143 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:53.006184 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:53.006778 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:53.007006 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:53.007010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:53.007784 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:53.008191 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.008199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.008996 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.008997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.010186 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.010189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.010671 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.010672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.010820 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.011342 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:53.011352 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:53.013154 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:53.015012 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:53.015019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:53.015021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:53.015255 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:53.015491 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:53.015492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:53.015492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:53.015610 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:53.015949 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:53.015954 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:53.016086 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 21:58:53.016177 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 21:58:53.016316 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260925 21:58:53.017009 18014 heartbeater.cc:347] Connected to a master server at 127.2.70.254:32829
I20260925 21:58:53.017098 18014 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:53.017254 18014 heartbeater.cc:511] Master 127.2.70.254:32829 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
I20260925 21:58:53.017658 17689 ts_manager.cc:194] Registered new tserver with Master: 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691)
I20260925 21:58:53.018186 17689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:50583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:58:53.035198 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:32829
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:58:53.116580 18019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:58:53.116802 18019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:58:53.116827 18019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:58:53.118046 18019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:58:53.118089 18019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:58:53.118117 18019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:58:53.119596 18019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:58:53.119849 18019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:58:53.120098 18019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:58:53.125062 18027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:53.125077 18025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:58:53.125257 18024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:58:53.125573 18019 server_base.cc:1034] running on GCE node
I20260925 21:58:53.125777 18019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:58:53.126073 18019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:58:53.127398 18019 hybrid_clock.cc:648] HybridClock initialized: now 1790373533127372 us; error 40 us; skew 500 ppm
Sep 25 21:58:53 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373533, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:58:53.130553 18019 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:58:53.131028 18019 webserver.cc:460] Webserver started at http://127.2.70.195:38087/ using document root <none> and password file <none>
I20260925 21:58:53.131289 18019 fs_manager.cc:359] Metadata directory not provided
I20260925 21:58:53.131356 18019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:58:53.131507 18019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:58:53.132670 18019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "4e39fa99a752489aa942336750eddef4"
format_stamp: "Formatted at 2026-09-25 21:58:53 on dist-test-slave-2g0f"
server_key: "109849c09d76632e0e541e251f3846fb"
server_key_iv: "ca9a44b13f6c9f079636c7631edb8058"
server_key_version: "encryptionkey@0"
I20260925 21:58:53.133003 18019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "4e39fa99a752489aa942336750eddef4"
format_stamp: "Formatted at 2026-09-25 21:58:53 on dist-test-slave-2g0f"
server_key: "109849c09d76632e0e541e251f3846fb"
server_key_iv: "ca9a44b13f6c9f079636c7631edb8058"
server_key_version: "encryptionkey@0"
I20260925 21:58:53.134637 18019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 21:58:53.135514 18034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:53.135704 18019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:58:53.135797 18019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "4e39fa99a752489aa942336750eddef4"
format_stamp: "Formatted at 2026-09-25 21:58:53 on dist-test-slave-2g0f"
server_key: "109849c09d76632e0e541e251f3846fb"
server_key_iv: "ca9a44b13f6c9f079636c7631edb8058"
server_key_version: "encryptionkey@0"
I20260925 21:58:53.135903 18019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:58:53.151384 18019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:58:53.153538 18019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:58:53.153685 18019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:58:53.153973 18019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:58:53.154295 18019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:58:53.154330 18019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:53.154358 18019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:58:53.154381 18019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:58:53.160058 18019 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:43127
I20260925 21:58:53.160101 18147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:43127 every 8 connection(s)
I20260925 21:58:53.160420 18019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
Sep 25 21:58:53 dist-test-slave-2g0f krb5kdc[17644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373533, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:58:53.169950 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.161143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:48701 (local address 127.2.70.254:32829)
0925 21:58:53.161194 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:53.161196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:53.161442 (+ 246us) server_negotiation.cc:407] Connection header received
0925 21:58:53.161939 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:53.161940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:53.161953 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:53.161999 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:53.163043 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.163553 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.164669 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.164872 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.164934 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.167104 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:53.167105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:53.167106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:53.167113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:53.168420 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:53.168832 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:53.168833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:53.168833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:53.168913 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:53.169205 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:53.169206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:53.169206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:53.169336 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:53.169398 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:53.169737 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 21:58:53.169820 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 21:58:53.169996 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 18019
I20260925 21:58:53.170087 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:58:53.170156 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.161136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:48701)
0925 21:58:53.161291 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:53.161332 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:53.161835 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:53.162115 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:53.162118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:53.162734 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:58:53.162943 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.162948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.163645 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.163646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.164537 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.164540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.164952 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.164953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.165076 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.165569 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:58:53.165577 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:58:53.167011 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:58:53.168497 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:53.168503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:53.168505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:53.168735 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:53.169004 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:58:53.169005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:58:53.169005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:58:53.169117 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:58:53.169427 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:58:53.169433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:58:53.169563 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 21:58:53.169664 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 21:58:53.169832 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260925 21:58:53.170285 2331 external_mini_cluster.cc:1428] Setting key 3ab263eab75c4904247e340f35126cd1
I20260925 21:58:53.170701 18148 heartbeater.cc:347] Connected to a master server at 127.2.70.254:32829
I20260925 21:58:53.170786 18148 heartbeater.cc:464] Registering TS with master...
I20260925 21:58:53.170941 18148 heartbeater.cc:511] Master 127.2.70.254:32829 requested a full tablet report, sending...
I20260925 21:58:53.171330 17689 ts_manager.cc:194] Registered new tserver with Master: 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127)
I20260925 21:58:53.171854 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:58:53.171867 17689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:48701
I20260925 21:58:53.179054 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.173315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.2.70.254:32829)
0925 21:58:53.173362 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:53.173363 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:53.173461 (+ 98us) server_negotiation.cc:407] Connection header received
0925 21:58:53.173503 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:53.173504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:53.173511 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:53.173559 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:58:53.174033 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.174514 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.175255 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.175436 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.175478 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.176646 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:58:53.176647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:58:53.176648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:58:53.176654 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:58:53.178005 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:53.178312 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:53.178313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:53.178314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:53.178375 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:58:53.178567 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:58:53.178567 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:58:53.178567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:58:53.178708 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:58:53.178809 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:53.178897 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 21:58:53.178979 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 21:58:53.180727 17689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:58:53.182430 17689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:58:53.188866 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.184904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.2.70.195:43127)
0925 21:58:53.185052 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:53.185057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:53.185067 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:58:53.185120 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:53.185122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:53.185138 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:53.185220 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:53.185516 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.186603 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.187800 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.188476 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.188565 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.188628 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:53.188746 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 21:58:53.188781 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260925 21:58:53.188858 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.184567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:43127 (local address 127.0.0.1:47938)
0925 21:58:53.184707 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:53.184733 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:53.184795 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:53.185224 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:53.185228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:53.185231 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:53.185302 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.185303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.186707 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.186708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.187728 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.187729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.188572 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.188574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.188658 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.188675 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 21:58:53.188726 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 21:58:53.188771 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":11,"thread_start_us":81,"threads_started":1}
I20260925 21:58:53.189067 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.184686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.2.70.193:33061)
0925 21:58:53.185140 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:53.185144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:53.185153 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:58:53.185189 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:53.185191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:53.185205 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:53.185272 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:53.185459 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.186252 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.187632 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.188381 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.188454 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.188524 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:53.188822 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 21:58:53.188872 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":91,"threads_started":1}
I20260925 21:58:53.188781 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.184565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:33061 (local address 127.0.0.1:50766)
0925 21:58:53.184693 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:53.184733 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:53.184793 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:53.185351 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:53.185352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:53.185354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:53.185405 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.185406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.186345 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.186346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.187555 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.187558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.188477 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.188478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.188558 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.188574 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 21:58:53.188635 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 21:58:53.188692 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":53,"threads_started":1}
I20260925 21:58:53.189702 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.184739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:35691 (local address 127.0.0.1:46334)
0925 21:58:53.184977 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:53.184986 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:53.185029 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:53.185330 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:53.185331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:53.185333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:53.185379 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.185381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.187793 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.187793 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.188554 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.188556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.189494 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.189495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.189563 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.189567 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:58:53.189594 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 21:58:53.189631 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":44,"threads_started":1}
I20260925 21:58:53.189615 18082 tablet_service.cc:1467] Processing CreateTablet for tablet bbd16221e3ff44118503a108da8f7bb7 (DEFAULT_TABLE table=test-table [id=33cf2251533e42ee949d5e81e2196175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:58:53.189832 17814 tablet_service.cc:1467] Processing CreateTablet for tablet bbd16221e3ff44118503a108da8f7bb7 (DEFAULT_TABLE table=test-table [id=33cf2251533e42ee949d5e81e2196175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:58:53.189980 18082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd16221e3ff44118503a108da8f7bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:53.190042 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.185000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.2.70.194:35691)
0925 21:58:53.185156 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:53.185161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:53.185170 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:58:53.185206 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:53.185208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:53.185227 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:53.185325 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:53.185493 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.186615 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.188647 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.189403 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.189812 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.189897 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:53.189924 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 21:58:53.189969 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260925 21:58:53.190171 17814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd16221e3ff44118503a108da8f7bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:53.192556 17948 tablet_service.cc:1467] Processing CreateTablet for tablet bbd16221e3ff44118503a108da8f7bb7 (DEFAULT_TABLE table=test-table [id=33cf2251533e42ee949d5e81e2196175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:58:53.192812 18162 tablet_bootstrap.cc:492] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4: Bootstrap starting.
I20260925 21:58:53.192899 17948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd16221e3ff44118503a108da8f7bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:58:53.193053 18163 tablet_bootstrap.cc:492] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Bootstrap starting.
I20260925 21:58:53.193672 18162 tablet_bootstrap.cc:654] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:53.193672 18163 tablet_bootstrap.cc:654] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:53.194257 18162 log.cc:824] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:53.194257 18163 log.cc:824] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:53.195017 18163 tablet_bootstrap.cc:492] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: No bootstrap required, opened a new log
I20260925 21:58:53.195086 18163 ts_tablet_manager.cc:1397] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:58:53.195242 18162 tablet_bootstrap.cc:492] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4: No bootstrap required, opened a new log
I20260925 21:58:53.195305 18162 ts_tablet_manager.cc:1397] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 21:58:53.195668 18166 tablet_bootstrap.cc:492] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8: Bootstrap starting.
I20260925 21:58:53.196271 18166 tablet_bootstrap.cc:654] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8: Neither blocks nor log segments found. Creating new log.
I20260925 21:58:53.196723 18166 log.cc:824] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8: Log is configured to *not* fsync() on all Append() calls
I20260925 21:58:53.197309 18166 tablet_bootstrap.cc:492] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8: No bootstrap required, opened a new log
I20260925 21:58:53.197185 18162 raft_consensus.cc:348] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } }
I20260925 21:58:53.197185 18163 raft_consensus.cc:348] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } }
I20260925 21:58:53.197361 18162 raft_consensus.cc:374] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:53.197365 18163 raft_consensus.cc:374] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:53.197365 18166 ts_tablet_manager.cc:1397] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:58:53.197407 18162 raft_consensus.cc:729] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e39fa99a752489aa942336750eddef4, State: Initialized, Role: FOLLOWER
I20260925 21:58:53.197407 18163 raft_consensus.cc:729] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e43ab7079304b4999f0f6927d2b3c0d, State: Initialized, Role: FOLLOWER
I20260925 21:58:53.197554 18162 consensus_queue.cc:260] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } }
I20260925 21:58:53.197554 18163 consensus_queue.cc:260] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } }
I20260925 21:58:53.197842 18162 ts_tablet_manager.cc:1428] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:58:53.197842 18163 ts_tablet_manager.cc:1428] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:58:53.197973 17880 heartbeater.cc:503] Master 127.2.70.254:32829 was elected leader, sending a full tablet report...
I20260925 21:58:53.198036 18148 heartbeater.cc:503] Master 127.2.70.254:32829 was elected leader, sending a full tablet report...
I20260925 21:58:53.199213 18166 raft_consensus.cc:348] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } }
I20260925 21:58:53.199353 18166 raft_consensus.cc:374] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:58:53.199394 18166 raft_consensus.cc:729] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1d335cbcbc459ca9eab6995926b1d8, State: Initialized, Role: FOLLOWER
I20260925 21:58:53.199509 18166 consensus_queue.cc:260] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } }
I20260925 21:58:53.199738 18166 ts_tablet_manager.cc:1428] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 21:58:53.199769 18014 heartbeater.cc:503] Master 127.2.70.254:32829 was elected leader, sending a full tablet report...
W20260925 21:58:53.255532 18015 tablet.cc:2369] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 21:58:53.362282 17881 tablet.cc:2369] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 21:58:53.411049 18149 tablet.cc:2369] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:58:53.431357 18168 raft_consensus.cc:482] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:58:53.431504 18168 raft_consensus.cc:504] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } }
I20260925 21:58:53.431872 18168 leader_election.cc:288] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e43ab7079304b4999f0f6927d2b3c0d (127.2.70.193:33061), 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691)
I20260925 21:58:53.432420 18169 raft_consensus.cc:482] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:58:53.432547 18169 raft_consensus.cc:504] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } }
I20260925 21:58:53.432861 18169 leader_election.cc:288] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127), 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691)
I20260925 21:58:53.435252 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.432130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:52091 (local address 127.2.70.193:33061)
0925 21:58:53.432176 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:53.432179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:53.432189 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:58:53.432225 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:53.432226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:53.432239 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:53.432298 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:53.432462 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.433121 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.434559 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.434936 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.434997 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.435006 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:53.435156 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 21:58:53.435193 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 21:58:53.435252 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.432047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:33061 (local address 127.2.70.195:52091)
0925 21:58:53.432112 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:53.432125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:53.432175 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:53.432321 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:53.432322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:53.432323 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:53.432396 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.432398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.433207 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.433208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.434457 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.434459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.435014 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.435015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.435092 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.435095 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:58:53.435140 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 21:58:53.435180 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260925 21:58:53.435518 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.432194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:37993 (local address 127.2.70.194:35691)
0925 21:58:53.432233 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:53.432235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:53.432334 (+ 99us) server_negotiation.cc:407] Connection header received
0925 21:58:53.432440 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:53.432442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:53.432454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:53.432502 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:53.432712 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.433376 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.434617 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.435168 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.435230 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.435237 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:53.435412 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 21:58:53.435451 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 21:58:53.435560 17834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd16221e3ff44118503a108da8f7bb7" candidate_uuid: "4e39fa99a752489aa942336750eddef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" is_pre_election: true
I20260925 21:58:53.435655 17834 raft_consensus.cc:2454] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e39fa99a752489aa942336750eddef4 in term 0.
I20260925 21:58:53.435736 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.432106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:35691 (local address 127.2.70.195:37993)
0925 21:58:53.432279 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:53.432287 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:53.432360 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:53.432548 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:53.432549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:53.432550 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:53.432620 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.432622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.433479 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.433480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.434547 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.434549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.435268 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.435269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.435338 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.435342 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:58:53.435624 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 21:58:53.435662 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":130,"threads_started":1}
I20260925 21:58:53.435878 18038 leader_election.cc:302] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e39fa99a752489aa942336750eddef4, 6e43ab7079304b4999f0f6927d2b3c0d; no voters:
I20260925 21:58:53.435990 18168 raft_consensus.cc:2781] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:58:53.436045 18168 raft_consensus.cc:482] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:58:53.436069 18168 raft_consensus.cc:3037] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:53.436334 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.433032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:34585 (local address 127.2.70.195:43127)
0925 21:58:53.433069 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:53.433071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:53.433178 (+ 107us) server_negotiation.cc:407] Connection header received
0925 21:58:53.433257 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:53.433258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:53.433265 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:53.433288 (+ 23us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:53.433467 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.434122 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.435462 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.435965 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.436002 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.436008 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:53.436249 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 21:58:53.436286 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":19}
I20260925 21:58:53.436264 17968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd16221e3ff44118503a108da8f7bb7" candidate_uuid: "4e39fa99a752489aa942336750eddef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" is_pre_election: true
I20260925 21:58:53.436448 17968 raft_consensus.cc:2454] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e39fa99a752489aa942336750eddef4 in term 0.
I20260925 21:58:53.436507 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.433016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:43127 (local address 127.2.70.193:34585)
0925 21:58:53.433122 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:53.433131 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:53.433199 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:53.433341 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:53.433341 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:53.433343 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:53.433399 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.433401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.434234 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.434234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.435384 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.435387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.436096 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.436097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.436168 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.436172 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:58:53.436393 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 21:58:53.436435 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260925 21:58:53.436797 18102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd16221e3ff44118503a108da8f7bb7" candidate_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e39fa99a752489aa942336750eddef4" is_pre_election: true
I20260925 21:58:53.436978 18168 raft_consensus.cc:504] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } }
I20260925 21:58:53.437124 18168 leader_election.cc:288] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e43ab7079304b4999f0f6927d2b3c0d (127.2.70.193:33061), 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691)
I20260925 21:58:53.437253 18102 raft_consensus.cc:2379] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e43ab7079304b4999f0f6927d2b3c0d in current term 1: Already voted for candidate 4e39fa99a752489aa942336750eddef4 in this term.
I20260925 21:58:53.437268 17834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd16221e3ff44118503a108da8f7bb7" candidate_uuid: "4e39fa99a752489aa942336750eddef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e43ab7079304b4999f0f6927d2b3c0d"
I20260925 21:58:53.437295 17968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd16221e3ff44118503a108da8f7bb7" candidate_uuid: "4e39fa99a752489aa942336750eddef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1d335cbcbc459ca9eab6995926b1d8"
I20260925 21:58:53.437332 17834 raft_consensus.cc:3037] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:53.437367 17968 raft_consensus.cc:3037] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:58:53.437997 17834 raft_consensus.cc:2454] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e39fa99a752489aa942336750eddef4 in term 1.
I20260925 21:58:53.438150 17968 raft_consensus.cc:2454] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e39fa99a752489aa942336750eddef4 in term 1.
I20260925 21:58:53.438150 18038 leader_election.cc:302] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e39fa99a752489aa942336750eddef4, 6e43ab7079304b4999f0f6927d2b3c0d; no voters:
I20260925 21:58:53.438306 18168 raft_consensus.cc:2781] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:58:53.438457 18168 raft_consensus.cc:686] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 1 LEADER]: Becoming Leader. State: Replica: 4e39fa99a752489aa942336750eddef4, State: Running, Role: LEADER
I20260925 21:58:53.438560 18168 consensus_queue.cc:237] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } }
I20260925 21:58:53.438882 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.433158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:35691 (local address 127.2.70.193:51263)
0925 21:58:53.433334 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:53.433343 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:58:53.433400 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:58:53.433763 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:58:53.433763 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:58:53.433765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:58:53.433819 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.433821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.435920 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.435921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.436900 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:58:53.436902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.438662 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.438663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:58:53.438722 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.438726 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:58:53.438770 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 21:58:53.438804 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":36,"threads_started":1}
I20260925 21:58:53.438891 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.433500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:51263 (local address 127.2.70.194:35691)
0925 21:58:53.433632 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:53.433633 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:53.433640 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:58:53.433659 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:53.433659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:53.433667 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:53.433723 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:58:53.433901 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.435796 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.436958 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.438555 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.438622 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.438631 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:53.438775 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 21:58:53.438808 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260925 21:58:53.439239 17968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd16221e3ff44118503a108da8f7bb7" candidate_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" is_pre_election: true
I20260925 21:58:53.439324 17968 raft_consensus.cc:2379] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e43ab7079304b4999f0f6927d2b3c0d in current term 1: Already voted for candidate 4e39fa99a752489aa942336750eddef4 in this term.
I20260925 21:58:53.439374 17689 catalog_manager.cc:5495] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e39fa99a752489aa942336750eddef4 (127.2.70.195). New cstate: current_term: 1 leader_uuid: "4e39fa99a752489aa942336750eddef4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } health_report { overall_health: UNKNOWN } } }
I20260925 21:58:53.439520 17767 leader_election.cc:302] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e43ab7079304b4999f0f6927d2b3c0d; no voters: 1d1d335cbcbc459ca9eab6995926b1d8, 4e39fa99a752489aa942336750eddef4
I20260925 21:58:53.439638 18169 raft_consensus.cc:2726] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 21:58:53.475067 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.472432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.2.70.195:43127)
0925 21:58:53.472495 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 21:58:53.472497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:58:53.472504 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:58:53.472539 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:58:53.472540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:58:53.472560 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:58:53.472620 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:58:53.472759 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.473249 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.474101 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:58:53.474313 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:58:53.474375 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:58:53.474552 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:58:53.474676 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:58:53.474752 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 21:58:53.474880 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 21:58:53.474923 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 21:58:53.477006 17968 raft_consensus.cc:1261] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 1 FOLLOWER]: Refusing update from remote peer 4e39fa99a752489aa942336750eddef4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:58:53.477164 17834 raft_consensus.cc:1261] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Refusing update from remote peer 4e39fa99a752489aa942336750eddef4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:58:53.477367 18168 consensus_queue.cc:1035] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:58:53.477504 18175 consensus_queue.cc:1035] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:58:53.480931 18180 mvcc.cc:205] Tried to move back new op lower bound from 7333369993118871552 to 7333369992964796416. Current Snapshot: MvccSnapshot[applied={T|T < 7333369993118871552}]
I20260925 21:58:53.482317 17689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:58:53.482414 17689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:58:53.483260 17689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Sending DeleteTablet for 3 replicas of tablet bbd16221e3ff44118503a108da8f7bb7
I20260925 21:58:53.483592 17814 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd16221e3ff44118503a108da8f7bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:53 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:50766
I20260925 21:58:53.483631 18082 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd16221e3ff44118503a108da8f7bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:53 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:47938
I20260925 21:58:53.483647 17948 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd16221e3ff44118503a108da8f7bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:53 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:46334
I20260925 21:58:53.483817 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 17751
I20260925 21:58:53.483940 18188 tablet_replica.cc:330] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: stopping tablet replica
I20260925 21:58:53.484036 18188 raft_consensus.cc:2229] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:58:53.484130 18188 raft_consensus.cc:2258] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:58:53.484347 18189 tablet_replica.cc:330] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4: stopping tablet replica
I20260925 21:58:53.484426 18189 raft_consensus.cc:2229] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:58:53.484504 18189 raft_consensus.cc:2258] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:58:53.484536 18190 tablet_replica.cc:330] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8: stopping tablet replica
I20260925 21:58:53.484627 18190 raft_consensus.cc:2229] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:58:53.484715 18190 raft_consensus.cc:2258] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:58:53.484977 18189 ts_tablet_manager.cc:1905] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:58:53.485086 18190 ts_tablet_manager.cc:1905] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:58:53.486737 18190 ts_tablet_manager.cc:1918] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:58:53.486804 18190 log.cc:1196] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/bbd16221e3ff44118503a108da8f7bb7
I20260925 21:58:53.486956 18189 ts_tablet_manager.cc:1918] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:58:53.487017 18189 log.cc:1196] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/bbd16221e3ff44118503a108da8f7bb7
I20260925 21:58:53.487265 18189 ts_tablet_manager.cc:1939] T bbd16221e3ff44118503a108da8f7bb7 P 4e39fa99a752489aa942336750eddef4: Deleting consensus metadata
I20260925 21:58:53.487552 18190 ts_tablet_manager.cc:1939] T bbd16221e3ff44118503a108da8f7bb7 P 1d1d335cbcbc459ca9eab6995926b1d8: Deleting consensus metadata
I20260925 21:58:53.487710 17673 catalog_manager.cc:4841] TS 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127): tablet bbd16221e3ff44118503a108da8f7bb7 (table test-table [id=33cf2251533e42ee949d5e81e2196175]) successfully deleted
I20260925 21:58:53.488109 17673 catalog_manager.cc:4841] TS 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691): tablet bbd16221e3ff44118503a108da8f7bb7 (table test-table [id=33cf2251533e42ee949d5e81e2196175]) successfully deleted
W20260925 21:58:53.490334 17676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:33061 (error 108)
I20260925 21:58:53.490514 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 17885
I20260925 21:58:53.491423 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:58:53.491285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:33061 (local address 127.0.0.1:50780)
0925 21:58:53.491331 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 21:58:53.491376 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 21:58:53.491561 17676 catalog_manager.cc:4584] TS 6e43ab7079304b4999f0f6927d2b3c0d (127.2.70.193:33061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bbd16221e3ff44118503a108da8f7bb7: Network error: Client connection negotiation failed: client connection to 127.2.70.193:33061: connect: Connection refused (error 111)
I20260925 21:58:53.494949 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 18019
I20260925 21:58:53.499024 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 17657
I20260925 21:59:13.503912 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:32829
--webserver_interface=127.2.70.254
--webserver_port=44703
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:13.586475 18192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:13.586869 18192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:13.586952 18192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:13.588730 18192 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 21:59:13.588788 18192 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 21:59:13.588801 18192 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 21:59:13.588814 18192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:59:13.588827 18192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:13.588840 18192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:13.588867 18192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:59:13.588891 18192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:59:13.590870 18192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:32829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=44703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:13.591177 18192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:13.591449 18192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:13.600145 18198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:13.600153 18199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:13.600153 18201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:13.600332 18192 server_base.cc:1034] running on GCE node
I20260925 21:59:13.600711 18192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:13.601054 18192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:13.602208 18192 hybrid_clock.cc:648] HybridClock initialized: now 1790373553602195 us; error 35 us; skew 500 ppm
Sep 25 21:59:13 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:13.610757 18192 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:59:13.611294 18192 webserver.cc:460] Webserver started at http://127.2.70.254:44703/ using document root <none> and password file <none>
I20260925 21:59:13.611541 18192 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:13.611609 18192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:13.613255 18192 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:13.613953 18208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:13.614169 18192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:13.614266 18192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "5ec1ce98a8614f5d8424eaea5faaa95d"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "ea4232ad2fd7daeebcd782d96ed02e8f"
server_key_iv: "56768b35d0bdd539a83f7eafdc52d848"
server_key_version: "encryptionkey@0"
I20260925 21:59:13.614737 18192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:13.628832 18192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:13.630718 18192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:13.630852 18192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:13.634729 18192 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:32829
I20260925 21:59:13.634765 18260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:32829 every 8 connection(s)
I20260925 21:59:13.635125 18192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:59:13.636615 18261 sys_catalog.cc:263] Verifying existing consensus state
I20260925 21:59:13.637406 18261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Bootstrap starting.
I20260925 21:59:13.639228 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 18192
I20260925 21:59:13.639498 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:33061
--local_ip_for_outbound_sockets=127.2.70.193
--tserver_master_addrs=127.2.70.254:32829
--webserver_port=36157
--webserver_interface=127.2.70.193
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:59:13.645802 18261 log.cc:824] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:13.647611 18261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 21:59:13.647892 18261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Bootstrap complete.
I20260925 21:59:13.649936 18261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } }
I20260925 21:59:13.650161 18261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec1ce98a8614f5d8424eaea5faaa95d, State: Initialized, Role: FOLLOWER
I20260925 21:59:13.650311 18261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } }
I20260925 21:59:13.650385 18261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:59:13.650437 18261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:59:13.650493 18261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 1 FOLLOWER]: Advancing to term 2
I20260925 21:59:13.651506 18261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } }
I20260925 21:59:13.651602 18261 leader_election.cc:302] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ec1ce98a8614f5d8424eaea5faaa95d; no voters:
I20260925 21:59:13.651780 18261 leader_election.cc:288] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 21:59:13.651866 18265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 21:59:13.652029 18265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 2 LEADER]: Becoming Leader. State: Replica: 5ec1ce98a8614f5d8424eaea5faaa95d, State: Running, Role: LEADER
I20260925 21:59:13.652046 18261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:59:13.652216 18265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } }
I20260925 21:59:13.652935 18266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ec1ce98a8614f5d8424eaea5faaa95d. Latest consensus state: current_term: 2 leader_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } } }
I20260925 21:59:13.653025 18266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:13.653204 18275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:59:13.653343 18267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } } }
I20260925 21:59:13.653430 18267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:13.654162 18275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:59:13.654260 18275 catalog_manager.cc:1259] Loaded cluster ID: 747b2058585c4b55baba0ff84a02e69a
I20260925 21:59:13.654524 18275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:59:13.655699 18275 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:59:13.655923 18275 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Loaded TSK: 0
I20260925 21:59:13.670686 18275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Generated new TSK 1
I20260925 21:59:13.670811 18275 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 21:59:13.724670 18263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:13.724866 18263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:13.724897 18263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:13.726056 18263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:13.726095 18263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:13.726130 18263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:59:13.727656 18263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:33061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=36157
--tserver_master_addrs=127.2.70.254:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:13.727903 18263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:13.728116 18263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:13.733146 18291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:13.733202 18289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:13.733251 18263 server_base.cc:1034] running on GCE node
W20260925 21:59:13.733376 18288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:13.733651 18263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:13.733886 18263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:13.735035 18263 hybrid_clock.cc:648] HybridClock initialized: now 1790373553735021 us; error 31 us; skew 500 ppm
Sep 25 21:59:13 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:13.737668 18263 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:59:13.738214 18263 webserver.cc:460] Webserver started at http://127.2.70.193:36157/ using document root <none> and password file <none>
I20260925 21:59:13.738523 18263 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:13.738611 18263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:13.740397 18263 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:59:13.741189 18298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:13.741351 18263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:13.741451 18263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "6e43ab7079304b4999f0f6927d2b3c0d"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "24f3f27d88442f2450199f598dd02ad6"
server_key_iv: "60e0d8153764eda7e9cdf6b5b56fd7a2"
server_key_version: "encryptionkey@0"
I20260925 21:59:13.741915 18263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:13.759982 18263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:13.762882 18263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:13.763053 18263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:13.763370 18263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:13.763912 18305 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 21:59:13.765470 18263 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 21:59:13.765538 18263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 21:59:13.765575 18263 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 21:59:13.766515 18263 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 21:59:13.766558 18263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 21:59:13.766657 18305 tablet_bootstrap.cc:492] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Bootstrap starting.
I20260925 21:59:13.773669 18263 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:33061
I20260925 21:59:13.774009 18412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:33061 every 8 connection(s)
I20260925 21:59:13.774160 18263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:59:13.774899 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 18263
I20260925 21:59:13.775205 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:35691
--local_ip_for_outbound_sockets=127.2.70.194
--tserver_master_addrs=127.2.70.254:32829
--webserver_port=45607
--webserver_interface=127.2.70.194
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:59:13.776954 18305 log.cc:824] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:13.779793 18305 tablet_bootstrap.cc:492] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 21:59:13.780149 18305 tablet_bootstrap.cc:492] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Bootstrap complete.
I20260925 21:59:13.781266 18305 ts_tablet_manager.cc:1397] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260925 21:59:13.783133 18305 raft_consensus.cc:348] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } }
I20260925 21:59:13.783540 18305 raft_consensus.cc:729] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e43ab7079304b4999f0f6927d2b3c0d, State: Initialized, Role: FOLLOWER
I20260925 21:59:13.783737 18305 consensus_queue.cc:260] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } }
I20260925 21:59:13.784081 18305 ts_tablet_manager.cc:1428] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
Sep 25 21:59:13 dist-test-slave-2g0f krb5kdc[17644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:13.789888 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:13.775093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:40917)
0925 21:59:13.775927 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:13.775996 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:13.776755 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:13.777479 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:13.777483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:13.777851 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:13.778127 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:13.778135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:13.779930 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:13.779932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:13.781233 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:13.781237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:13.782228 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:13.782230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:13.782403 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:13.783074 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:13.783094 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:13.785156 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:13.787769 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:13.787779 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:13.787781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:13.788193 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:13.788546 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:13.788547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:13.788547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:13.788689 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:13.789205 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:13.789211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:13.789379 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 21:59:13.789481 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 21:59:13.789665 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":112,"threads_started":1}
I20260925 21:59:13.790196 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:13.775147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:40917 (local address 127.2.70.254:32829)
0925 21:59:13.775757 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:13.775768 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:13.776970 (+ 1202us) server_negotiation.cc:407] Connection header received
0925 21:59:13.777125 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:13.777132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:13.777200 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:13.777346 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:13.778283 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:13.779741 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:13.781414 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:13.782073 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:13.782724 (+ 651us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:13.785296 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:13.785302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:13.785302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:13.785318 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:13.787654 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:13.788311 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:13.788315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:13.788316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:13.788468 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:13.788773 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:13.788774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:13.788775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:13.789118 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:13.789216 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:13.789582 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 21:59:13.789741 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":90,"threads_started":1}
I20260925 21:59:13.791108 18413 heartbeater.cc:347] Connected to a master server at 127.2.70.254:32829
I20260925 21:59:13.791241 18413 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:13.791457 18413 heartbeater.cc:511] Master 127.2.70.254:32829 requested a full tablet report, sending...
I20260925 21:59:13.792440 18225 ts_manager.cc:194] Registered new tserver with Master: 6e43ab7079304b4999f0f6927d2b3c0d (127.2.70.193:33061)
I20260925 21:59:13.793730 18225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:40917
I20260925 21:59:13.794818 18413 heartbeater.cc:503] Master 127.2.70.254:32829 was elected leader, sending a full tablet report...
I20260925 21:59:13.800437 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:13.793416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.2.70.193:33061)
0925 21:59:13.796198 (+ 2782us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:13.796203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:13.796211 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:59:13.796244 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:13.796246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:13.796261 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:13.796328 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:13.796675 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:13.797773 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:13.799324 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:13.800010 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:13.800076 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:13.800144 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:13.800315 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 21:59:13.800358 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2748,"thread_start_us":78,"threads_started":1}
I20260925 21:59:13.800454 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:13.793140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:33061 (local address 127.0.0.1:35460)
0925 21:59:13.795948 (+ 2808us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:13.795990 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:13.796060 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:13.796523 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:13.796526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:13.796529 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:13.796601 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:13.796604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:13.797862 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:13.797863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:13.799219 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:13.799227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:13.800109 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:13.800110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:13.800209 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:13.800225 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 21:59:13.800296 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 21:59:13.800354 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2772,"thread_start_us":104,"threads_started":1}
I20260925 21:59:13.801154 18347 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd16221e3ff44118503a108da8f7bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:58:53 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:35460
I20260925 21:59:13.801782 18422 tablet_replica.cc:330] stopping tablet replica
I20260925 21:59:13.801908 18422 raft_consensus.cc:2229] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:59:13.801982 18422 raft_consensus.cc:2258] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:13.802333 18422 ts_tablet_manager.cc:1905] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:13.805195 18422 ts_tablet_manager.cc:1918] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:13.805289 18422 log.cc:1196] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal/wals/bbd16221e3ff44118503a108da8f7bb7
I20260925 21:59:13.805500 18422 ts_tablet_manager.cc:1939] T bbd16221e3ff44118503a108da8f7bb7 P 6e43ab7079304b4999f0f6927d2b3c0d: Deleting consensus metadata
I20260925 21:59:13.806165 18212 catalog_manager.cc:4841] TS 6e43ab7079304b4999f0f6927d2b3c0d (127.2.70.193:33061): tablet bbd16221e3ff44118503a108da8f7bb7 (table test-table [id=33cf2251533e42ee949d5e81e2196175]) successfully deleted
W20260925 21:59:13.867381 18417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:13.867609 18417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:13.867633 18417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:13.868849 18417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:13.868953 18417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:13.868983 18417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:59:13.870513 18417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:35691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=45607
--tserver_master_addrs=127.2.70.254:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:13.870751 18417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:13.870958 18417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:13.876227 18427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:13.876241 18430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:13.876250 18428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:13.876461 18417 server_base.cc:1034] running on GCE node
I20260925 21:59:13.876679 18417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:13.876943 18417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:13.878118 18417 hybrid_clock.cc:648] HybridClock initialized: now 1790373553878103 us; error 28 us; skew 500 ppm
Sep 25 21:59:13 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:13.880735 18417 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:59:13.881209 18417 webserver.cc:460] Webserver started at http://127.2.70.194:45607/ using document root <none> and password file <none>
I20260925 21:59:13.881430 18417 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:13.881515 18417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:13.882874 18417 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:13.883522 18437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:13.883685 18417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:13.883767 18417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "1d1d335cbcbc459ca9eab6995926b1d8"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "ebd36810ec28cec268ac367f87f362e4"
server_key_iv: "1f8ef791e75cc9e81df39baa9a8e274d"
server_key_version: "encryptionkey@0"
I20260925 21:59:13.884199 18417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:13.923134 18417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:13.925426 18417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:13.925567 18417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:13.925783 18417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:13.926103 18417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:13.926137 18417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:13.926167 18417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:13.926190 18417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:13.931630 18417 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:35691
I20260925 21:59:13.931690 18550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:35691 every 8 connection(s)
I20260925 21:59:13.932061 18417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:59:13.936252 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 18417
I20260925 21:59:13.936532 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:43127
--local_ip_for_outbound_sockets=127.2.70.195
--tserver_master_addrs=127.2.70.254:32829
--webserver_port=38087
--webserver_interface=127.2.70.195
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 21:59:13 dist-test-slave-2g0f krb5kdc[17644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:13.941619 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:13.932819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:50563 (local address 127.2.70.254:32829)
0925 21:59:13.932900 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:13.932901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:13.933066 (+ 165us) server_negotiation.cc:407] Connection header received
0925 21:59:13.933541 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:13.933542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:13.933556 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:13.933618 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:13.934569 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:13.935083 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:13.936184 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:13.936403 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:13.936458 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:13.938734 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:13.938736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:13.938736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:13.938743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:13.940080 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:13.940445 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:13.940446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:13.940446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:13.940499 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:13.940751 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:13.940752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:13.940752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:13.940929 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:13.940999 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:13.941349 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 21:59:13.941454 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 21:59:13.941686 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:13.932789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:50563)
0925 21:59:13.932942 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:13.932985 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:13.933456 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:13.933672 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:13.933676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:13.934250 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:13.934454 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:13.934460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:13.935178 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:13.935179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:13.936105 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:13.936108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:13.936480 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:13.936481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:13.936626 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:13.937107 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:13.937117 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:13.938640 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:13.940152 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:13.940159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:13.940161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:13.940379 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:13.940568 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:13.940569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:13.940569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:13.940693 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:13.941027 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:13.941034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:13.941188 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 21:59:13.941287 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 21:59:13.941456 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260925 21:59:13.942080 18551 heartbeater.cc:347] Connected to a master server at 127.2.70.254:32829
I20260925 21:59:13.942171 18551 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:13.942342 18551 heartbeater.cc:511] Master 127.2.70.254:32829 requested a full tablet report, sending...
I20260925 21:59:13.942700 18225 ts_manager.cc:194] Registered new tserver with Master: 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691)
I20260925 21:59:13.943066 18225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:50563
W20260925 21:59:14.019209 18554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:14.019435 18554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:14.019469 18554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:14.020673 18554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:14.020710 18554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:14.020744 18554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:59:14.022564 18554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:43127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=38087
--tserver_master_addrs=127.2.70.254:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:14.022791 18554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:14.022994 18554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:14.027815 18562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:14.027818 18560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:14.028052 18554 server_base.cc:1034] running on GCE node
W20260925 21:59:14.028004 18559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:14.028285 18554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:14.028522 18554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:14.029670 18554 hybrid_clock.cc:648] HybridClock initialized: now 1790373554029652 us; error 32 us; skew 500 ppm
Sep 25 21:59:14 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373554, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:14.032018 18554 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:59:14.032431 18554 webserver.cc:460] Webserver started at http://127.2.70.195:38087/ using document root <none> and password file <none>
I20260925 21:59:14.032629 18554 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:14.032686 18554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:14.033953 18554 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:14.034463 18569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:14.034590 18554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:14.034668 18554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "4e39fa99a752489aa942336750eddef4"
format_stamp: "Formatted at 2026-09-25 21:58:53 on dist-test-slave-2g0f"
server_key: "109849c09d76632e0e541e251f3846fb"
server_key_iv: "ca9a44b13f6c9f079636c7631edb8058"
server_key_version: "encryptionkey@0"
I20260925 21:59:14.034986 18554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:14.062232 18554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:14.064255 18554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:14.064397 18554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:14.064647 18554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:14.064944 18554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:14.064978 18554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:14.065009 18554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:14.065026 18554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:14.070765 18554 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:43127
I20260925 21:59:14.070819 18682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:43127 every 8 connection(s)
I20260925 21:59:14.071172 18554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:59:14.072148 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 18554
Sep 25 21:59:14 dist-test-slave-2g0f krb5kdc[17644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373554, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:14.079038 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.072567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48712 (local address 127.2.70.254:32829)
0925 21:59:14.072676 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:14.072678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:14.072721 (+ 43us) server_negotiation.cc:407] Connection header received
0925 21:59:14.072792 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:14.072793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:14.072804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:14.072858 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:14.073463 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.073972 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.074718 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.074923 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.074977 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.076404 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:14.076405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:14.076406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:14.076412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:14.077839 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:14.078165 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:14.078166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:14.078166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:14.078246 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:14.078434 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:14.078435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:14.078435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:14.078571 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:14.078691 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:14.078795 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 21:59:14.078883 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260925 21:59:14.080721 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.071946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:59891 (local address 127.2.70.254:32829)
0925 21:59:14.072085 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:14.072087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:14.072268 (+ 181us) server_negotiation.cc:407] Connection header received
0925 21:59:14.072893 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:14.072894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:14.072904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:14.072969 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:14.073926 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.074409 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.075472 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.075706 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.075778 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.078149 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:14.078150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:14.078150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:14.078156 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:14.079387 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:14.079769 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:14.079770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:14.079770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:14.079824 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:14.080110 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:14.080111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:14.080112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:14.080223 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:14.080278 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:14.080543 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 21:59:14.080632 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 21:59:14.080920 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.071954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:59891)
0925 21:59:14.072126 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:14.072179 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:14.072797 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:14.073084 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:14.073088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:14.073639 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:14.073835 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:14.073839 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.074508 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.074509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:14.075392 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:14.075395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.075795 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.075796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:14.075944 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.076443 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:14.076451 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:14.078067 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:14.079452 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:14.079484 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:14.079485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:14.079702 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:14.079912 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:14.079913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:14.079913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:14.080040 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:14.080277 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:14.080284 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:14.080414 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 21:59:14.080513 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 21:59:14.080684 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260925 21:59:14.081372 18683 heartbeater.cc:347] Connected to a master server at 127.2.70.254:32829
I20260925 21:59:14.081465 18683 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:14.081662 18683 heartbeater.cc:511] Master 127.2.70.254:32829 requested a full tablet report, sending...
I20260925 21:59:14.082028 18225 ts_manager.cc:194] Registered new tserver with Master: 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127)
I20260925 21:59:14.082401 18225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:59891
I20260925 21:59:14.091166 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:59:14.094729 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.091677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.2.70.254:32829)
0925 21:59:14.091740 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:14.091742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:14.091871 (+ 129us) server_negotiation.cc:407] Connection header received
0925 21:59:14.092020 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:14.092021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:14.092064 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:14.092145 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:59:14.092405 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.092955 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.093773 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.093999 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.094062 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.094316 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:59:14.094367 (+ 51us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 21:59:14.094651 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48722: token expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 21:59:14.094749 18153 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.70.254:32829)
W20260925 21:59:14.094811 18416 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48722: token expired
I20260925 21:59:14.102519 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.095116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.2.70.254:32829)
0925 21:59:14.095166 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:14.095168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:14.095267 (+ 99us) server_negotiation.cc:407] Connection header received
0925 21:59:14.095329 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:14.095330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:14.095338 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:14.095382 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:14.095990 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.096728 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.097785 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.098135 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.098207 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.099692 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:14.099693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:14.099693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:14.099703 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:14.101273 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:14.101631 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:14.101632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:14.101633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:14.101695 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:14.101914 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:14.101915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:14.101915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:14.102132 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:14.102232 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:14.102328 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 21:59:14.102416 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 21:59:14.103989 18225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:59:14.105721 18225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:59:14.107957 18347 tablet_service.cc:1467] Processing CreateTablet for tablet f0a6dbbda52f4932804722cfd487f6e4 (DEFAULT_TABLE table=test-table [id=1bb7a7900e864d0da626af106c1abb6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:14.108186 18347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a6dbbda52f4932804722cfd487f6e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:14.109573 18305 tablet_bootstrap.cc:492] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: Bootstrap starting.
I20260925 21:59:14.109901 18305 tablet_bootstrap.cc:654] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:14.110409 18305 tablet_bootstrap.cc:492] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: No bootstrap required, opened a new log
I20260925 21:59:14.110443 18305 ts_tablet_manager.cc:1397] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:14.110533 18305 raft_consensus.cc:348] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:14.110582 18305 raft_consensus.cc:374] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:14.110630 18305 raft_consensus.cc:729] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e43ab7079304b4999f0f6927d2b3c0d, State: Initialized, Role: FOLLOWER
I20260925 21:59:14.110671 18305 consensus_queue.cc:260] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:14.110761 18305 ts_tablet_manager.cc:1428] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:14.112365 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.107625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:35691 (local address 127.0.0.1:54062)
0925 21:59:14.107889 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:14.107901 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:14.107943 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:14.108463 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:14.108463 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:14.108465 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:14.108519 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:14.108521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.109911 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.109912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:14.111087 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:14.111091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.112146 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.112147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:14.112222 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.112225 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:14.112264 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 21:59:14.112307 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260925 21:59:14.112506 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.108083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.2.70.194:35691)
0925 21:59:14.108313 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:14.108318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:14.108329 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:59:14.108373 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:14.108375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:14.108393 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:14.108468 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:14.108582 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.109823 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.111254 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.112064 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.112146 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.112215 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:14.112292 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 21:59:14.112339 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260925 21:59:14.112499 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.107727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:43127 (local address 127.0.0.1:60192)
0925 21:59:14.108279 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:14.108290 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:14.108347 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:14.108618 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:14.108619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:14.108621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:14.108668 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:14.108669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.110016 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.110017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:14.111148 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:14.111150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.112252 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.112253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:14.112325 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.112329 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:59:14.112381 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 21:59:14.112426 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":504,"threads_started":1}
I20260925 21:59:14.112576 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.107855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.2.70.195:43127)
0925 21:59:14.108338 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:14.108343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:14.108355 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:59:14.108421 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:14.108423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:14.108439 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:14.108517 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:14.108748 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.109940 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.111240 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.112100 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.112154 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.112215 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:14.112414 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 21:59:14.112459 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":119,"threads_started":1}
I20260925 21:59:14.113389 18617 tablet_service.cc:1467] Processing CreateTablet for tablet f0a6dbbda52f4932804722cfd487f6e4 (DEFAULT_TABLE table=test-table [id=1bb7a7900e864d0da626af106c1abb6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:14.113678 18617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a6dbbda52f4932804722cfd487f6e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:14.116547 18693 tablet_bootstrap.cc:492] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4: Bootstrap starting.
I20260925 21:59:14.116942 18485 tablet_service.cc:1467] Processing CreateTablet for tablet f0a6dbbda52f4932804722cfd487f6e4 (DEFAULT_TABLE table=test-table [id=1bb7a7900e864d0da626af106c1abb6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:14.117183 18693 tablet_bootstrap.cc:654] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:14.117283 18485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a6dbbda52f4932804722cfd487f6e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:14.117908 18693 log.cc:824] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:14.118564 18693 tablet_bootstrap.cc:492] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4: No bootstrap required, opened a new log
I20260925 21:59:14.118623 18693 ts_tablet_manager.cc:1397] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:59:14.120000 18695 tablet_bootstrap.cc:492] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8: Bootstrap starting.
I20260925 21:59:14.120074 18693 raft_consensus.cc:348] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:14.120246 18693 raft_consensus.cc:374] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:14.120289 18693 raft_consensus.cc:729] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e39fa99a752489aa942336750eddef4, State: Initialized, Role: FOLLOWER
I20260925 21:59:14.120407 18693 consensus_queue.cc:260] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:14.120630 18695 tablet_bootstrap.cc:654] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:14.120671 18693 ts_tablet_manager.cc:1428] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:59:14.120680 18683 heartbeater.cc:503] Master 127.2.70.254:32829 was elected leader, sending a full tablet report...
I20260925 21:59:14.121137 18695 log.cc:824] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:14.121876 18695 tablet_bootstrap.cc:492] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8: No bootstrap required, opened a new log
I20260925 21:59:14.121939 18695 ts_tablet_manager.cc:1397] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:59:14.123206 18695 raft_consensus.cc:348] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:14.123339 18695 raft_consensus.cc:374] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:14.123369 18695 raft_consensus.cc:729] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1d335cbcbc459ca9eab6995926b1d8, State: Initialized, Role: FOLLOWER
I20260925 21:59:14.123487 18695 consensus_queue.cc:260] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:14.123709 18695 ts_tablet_manager.cc:1428] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:59:14.123744 18551 heartbeater.cc:503] Master 127.2.70.254:32829 was elected leader, sending a full tablet report...
I20260925 21:59:14.129911 18419 raft_consensus.cc:482] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:59:14.130030 18419 raft_consensus.cc:504] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:14.130303 18419 leader_election.cc:288] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691), 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127)
I20260925 21:59:14.133391 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.130570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:60241 (local address 127.2.70.194:35691)
0925 21:59:14.130642 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:14.130644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:14.130652 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:59:14.130674 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:14.130675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:14.130686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:14.130729 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:14.130916 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.131645 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.132713 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.133092 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.133157 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.133167 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:14.133291 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 21:59:14.133328 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 21:59:14.133394 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.130466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:35691 (local address 127.2.70.193:60241)
0925 21:59:14.130503 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:14.130515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:14.130562 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:14.130779 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:14.130780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:14.130782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:14.130851 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:14.130853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.131774 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.131775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:14.132626 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:14.132629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.133167 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.133167 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:14.133233 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.133236 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:14.133280 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 21:59:14.133324 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 21:59:14.133448 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.130702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:45609 (local address 127.2.70.195:43127)
0925 21:59:14.130759 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:14.130761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:14.130786 (+ 25us) server_negotiation.cc:407] Connection header received
0925 21:59:14.130845 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:14.130846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:14.130853 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:14.130901 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:14.131092 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.131796 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.132780 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.133152 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.133210 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.133217 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:14.133353 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 21:59:14.133387 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 21:59:14.133478 18699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.130599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:43127 (local address 127.2.70.193:45609)
0925 21:59:14.130738 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:14.130747 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:14.130796 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:14.130969 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:14.130970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:14.130971 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:14.131030 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:14.131032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.131890 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.131891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:14.132700 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:14.132703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.133237 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.133238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:14.133297 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.133300 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:14.133343 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 21:59:14.133381 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260925 21:59:14.133786 18637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a6dbbda52f4932804722cfd487f6e4" candidate_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e39fa99a752489aa942336750eddef4" is_pre_election: true
I20260925 21:59:14.133816 18505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a6dbbda52f4932804722cfd487f6e4" candidate_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" is_pre_election: true
I20260925 21:59:14.133937 18637 raft_consensus.cc:2454] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e43ab7079304b4999f0f6927d2b3c0d in term 0.
I20260925 21:59:14.133937 18505 raft_consensus.cc:2454] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e43ab7079304b4999f0f6927d2b3c0d in term 0.
I20260925 21:59:14.134124 18299 leader_election.cc:302] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d1d335cbcbc459ca9eab6995926b1d8, 6e43ab7079304b4999f0f6927d2b3c0d; no voters:
I20260925 21:59:14.134231 18419 raft_consensus.cc:2781] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:59:14.134284 18419 raft_consensus.cc:482] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:59:14.134303 18419 raft_consensus.cc:3037] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:14.134902 18419 raft_consensus.cc:504] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:14.135013 18419 leader_election.cc:288] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 1 election: Requested vote from peers 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691), 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127)
I20260925 21:59:14.135170 18505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a6dbbda52f4932804722cfd487f6e4" candidate_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1d335cbcbc459ca9eab6995926b1d8"
I20260925 21:59:14.135205 18637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a6dbbda52f4932804722cfd487f6e4" candidate_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e39fa99a752489aa942336750eddef4"
I20260925 21:59:14.135241 18505 raft_consensus.cc:3037] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:14.135277 18637 raft_consensus.cc:3037] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:14.135895 18505 raft_consensus.cc:2454] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e43ab7079304b4999f0f6927d2b3c0d in term 1.
I20260925 21:59:14.136026 18637 raft_consensus.cc:2454] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e43ab7079304b4999f0f6927d2b3c0d in term 1.
I20260925 21:59:14.136057 18299 leader_election.cc:302] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d1d335cbcbc459ca9eab6995926b1d8, 6e43ab7079304b4999f0f6927d2b3c0d; no voters:
I20260925 21:59:14.136124 18419 raft_consensus.cc:2781] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:14.136276 18419 raft_consensus.cc:686] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 LEADER]: Becoming Leader. State: Replica: 6e43ab7079304b4999f0f6927d2b3c0d, State: Running, Role: LEADER
I20260925 21:59:14.136343 18419 consensus_queue.cc:237] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:14.137032 18224 catalog_manager.cc:5495] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e43ab7079304b4999f0f6927d2b3c0d (127.2.70.193). New cstate: current_term: 1 leader_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } health_report { overall_health: HEALTHY } } }
I20260925 21:59:14.148973 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.146944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.2.70.193:33061)
0925 21:59:14.147025 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:14.147027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:14.147034 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:59:14.147068 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:14.147069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:14.147092 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:14.147156 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:59:14.147251 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.147699 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.148369 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:14.148554 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:14.148608 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:14.148709 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:59:14.148802 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:59:14.148850 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:14.148881 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 21:59:14.148900 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 21:59:14.150506 18637 raft_consensus.cc:1261] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4 [term 1 FOLLOWER]: Refusing update from remote peer 6e43ab7079304b4999f0f6927d2b3c0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:14.150537 18505 raft_consensus.cc:1261] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 1 FOLLOWER]: Refusing update from remote peer 6e43ab7079304b4999f0f6927d2b3c0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:14.150830 18419 consensus_queue.cc:1035] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:14.150991 18700 consensus_queue.cc:1035] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:14.153959 18704 mvcc.cc:205] Tried to move back new op lower bound from 7333370077798080512 to 7333370077743017984. Current Snapshot: MvccSnapshot[applied={T|T < 7333370077798080512}]
I20260925 21:59:14.153995 18705 mvcc.cc:205] Tried to move back new op lower bound from 7333370077798080512 to 7333370077743017984. Current Snapshot: MvccSnapshot[applied={T|T < 7333370077798080512}]
I20260925 21:59:14.156065 18225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:14.156178 18225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:14.156893 18225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Sending DeleteTablet for 3 replicas of tablet f0a6dbbda52f4932804722cfd487f6e4
I20260925 21:59:14.157183 18347 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a6dbbda52f4932804722cfd487f6e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:14 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:35460
I20260925 21:59:14.157276 18422 tablet_replica.cc:330] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: stopping tablet replica
I20260925 21:59:14.157316 18422 raft_consensus.cc:2229] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:59:14.157303 18617 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a6dbbda52f4932804722cfd487f6e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:14 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:60192
I20260925 21:59:14.157305 18485 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a6dbbda52f4932804722cfd487f6e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:14 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:54062
I20260925 21:59:14.157395 18422 raft_consensus.cc:2258] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:14.157557 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 18263
I20260925 21:59:14.157735 18422 ts_tablet_manager.cc:1905] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:14.158597 18714 tablet_replica.cc:330] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8: stopping tablet replica
I20260925 21:59:14.158700 18714 raft_consensus.cc:2229] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:59:14.158808 18714 raft_consensus.cc:2258] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:14.159343 18714 ts_tablet_manager.cc:1905] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:14.159962 18713 tablet_replica.cc:330] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4: stopping tablet replica
I20260925 21:59:14.160079 18713 raft_consensus.cc:2229] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:59:14.160192 18713 raft_consensus.cc:2258] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:14.161434 18714 ts_tablet_manager.cc:1918] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:14.161516 18714 log.cc:1196] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/f0a6dbbda52f4932804722cfd487f6e4
I20260925 21:59:14.161787 18714 ts_tablet_manager.cc:1939] T f0a6dbbda52f4932804722cfd487f6e4 P 1d1d335cbcbc459ca9eab6995926b1d8: Deleting consensus metadata
I20260925 21:59:14.162263 18209 catalog_manager.cc:4841] TS 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691): tablet f0a6dbbda52f4932804722cfd487f6e4 (table test-table [id=1bb7a7900e864d0da626af106c1abb6c]) successfully deleted
W20260925 21:59:14.163113 18212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:33061 (error 108)
I20260925 21:59:14.163720 18713 ts_tablet_manager.cc:1905] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:14.163753 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 18417
I20260925 21:59:14.163787 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:14.163630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:33061 (local address 127.0.0.1:35466)
0925 21:59:14.163701 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:14.163751 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 21:59:14.163932 18212 catalog_manager.cc:4584] TS 6e43ab7079304b4999f0f6927d2b3c0d (127.2.70.193:33061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0a6dbbda52f4932804722cfd487f6e4: Network error: Client connection negotiation failed: client connection to 127.2.70.193:33061: connect: Connection refused (error 111)
I20260925 21:59:14.166625 18713 ts_tablet_manager.cc:1918] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:14.166700 18713 log.cc:1196] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/f0a6dbbda52f4932804722cfd487f6e4
I20260925 21:59:14.167013 18713 ts_tablet_manager.cc:1939] T f0a6dbbda52f4932804722cfd487f6e4 P 4e39fa99a752489aa942336750eddef4: Deleting consensus metadata
I20260925 21:59:14.167449 18209 catalog_manager.cc:4841] TS 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127): tablet f0a6dbbda52f4932804722cfd487f6e4 (table test-table [id=1bb7a7900e864d0da626af106c1abb6c]) successfully deleted
I20260925 21:59:14.168248 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 18554
I20260925 21:59:14.172555 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 18192
I20260925 21:59:34.178020 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:32829
--webserver_interface=127.2.70.254
--webserver_port=44703
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:34.259008 18716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:34.259248 18716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:34.259271 18716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:34.260505 18716 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 21:59:34.260548 18716 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 21:59:34.260560 18716 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 21:59:34.260571 18716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:59:34.260582 18716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:34.260593 18716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:34.260609 18716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:59:34.260633 18716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:59:34.262158 18716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:32829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=44703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:34.262398 18716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:34.262619 18716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:59:34.267864 18716 server_base.cc:1034] running on GCE node
W20260925 21:59:34.267798 18722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:34.267783 18721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:34.267791 18724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:34.268349 18716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:34.268617 18716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:34.269762 18716 hybrid_clock.cc:648] HybridClock initialized: now 1790373574269750 us; error 31 us; skew 500 ppm
Sep 25 21:59:34 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373574, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:34.277910 18716 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:59:34.278406 18716 webserver.cc:460] Webserver started at http://127.2.70.254:44703/ using document root <none> and password file <none>
I20260925 21:59:34.278628 18716 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:34.278674 18716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:34.280049 18716 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:59:34.280681 18731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:34.280833 18716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:34.280917 18716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "5ec1ce98a8614f5d8424eaea5faaa95d"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "ea4232ad2fd7daeebcd782d96ed02e8f"
server_key_iv: "56768b35d0bdd539a83f7eafdc52d848"
server_key_version: "encryptionkey@0"
I20260925 21:59:34.281314 18716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:34.297544 18716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:34.299679 18716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:34.299805 18716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:34.303764 18716 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:32829
I20260925 21:59:34.303805 18783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:32829 every 8 connection(s)
I20260925 21:59:34.304209 18716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:59:34.305693 18784 sys_catalog.cc:263] Verifying existing consensus state
I20260925 21:59:34.306435 18784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Bootstrap starting.
I20260925 21:59:34.313320 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 18716
I20260925 21:59:34.313616 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:33061
--local_ip_for_outbound_sockets=127.2.70.193
--tserver_master_addrs=127.2.70.254:32829
--webserver_port=36157
--webserver_interface=127.2.70.193
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:59:34.314704 18784 log.cc:824] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:34.317062 18784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 21:59:34.317341 18784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: Bootstrap complete.
I20260925 21:59:34.319413 18784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } }
I20260925 21:59:34.319638 18784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec1ce98a8614f5d8424eaea5faaa95d, State: Initialized, Role: FOLLOWER
I20260925 21:59:34.319790 18784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } }
I20260925 21:59:34.319891 18784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:59:34.319947 18784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:59:34.320001 18784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 2 FOLLOWER]: Advancing to term 3
I20260925 21:59:34.320994 18784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } }
I20260925 21:59:34.321094 18784 leader_election.cc:302] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ec1ce98a8614f5d8424eaea5faaa95d; no voters:
I20260925 21:59:34.321264 18784 leader_election.cc:288] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 21:59:34.321342 18788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 3 FOLLOWER]: Leader election won for term 3
I20260925 21:59:34.321506 18788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [term 3 LEADER]: Becoming Leader. State: Replica: 5ec1ce98a8614f5d8424eaea5faaa95d, State: Running, Role: LEADER
I20260925 21:59:34.321525 18784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:59:34.321614 18788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } }
I20260925 21:59:34.321933 18789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } } }
I20260925 21:59:34.321967 18788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ec1ce98a8614f5d8424eaea5faaa95d. Latest consensus state: current_term: 3 leader_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec1ce98a8614f5d8424eaea5faaa95d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 32829 } } }
I20260925 21:59:34.322021 18789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:34.322031 18788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:34.323278 18804 catalog_manager.cc:1259] Loaded cluster ID: 747b2058585c4b55baba0ff84a02e69a
I20260925 21:59:34.323328 18804 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: loading cluster ID for follower catalog manager: success
I20260925 21:59:34.324501 18804 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ec1ce98a8614f5d8424eaea5faaa95d: acquiring CA information for follower catalog manager: success
F20260925 21:59:34.324877 18804 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd99b8999ad google::LogMessage::Fail() at ??:0
@ 0x7fd99b89c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd99b8994de google::LogMessage::Flush() at ??:0
@ 0x7fd99b89aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd99e006104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd9a37f66b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fd9a37f6989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fd9a37f6e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fd9a37f98c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fd99d75c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd99e5c06db start_thread at ??:0
@ 0x7fd99a39761f clone at ??:0
W20260925 21:59:34.407153 18786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:34.407358 18786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:34.407382 18786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:34.408641 18786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:34.408687 18786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:34.408713 18786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:59:34.410198 18786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:33061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=36157
--tserver_master_addrs=127.2.70.254:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:34.410426 18786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:34.410666 18786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:34.415405 18822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:34.415412 18823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:34.415500 18825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:34.415863 18786 server_base.cc:1034] running on GCE node
I20260925 21:59:34.416054 18786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:34.416337 18786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:34.417480 18786 hybrid_clock.cc:648] HybridClock initialized: now 1790373574417470 us; error 25 us; skew 500 ppm
Sep 25 21:59:34 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373574, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:34.419973 18786 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:59:34.420465 18786 webserver.cc:460] Webserver started at http://127.2.70.193:36157/ using document root <none> and password file <none>
I20260925 21:59:34.420670 18786 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:34.420712 18786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:34.421985 18786 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:34.422672 18832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:34.422832 18786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:34.422933 18786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "6e43ab7079304b4999f0f6927d2b3c0d"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "24f3f27d88442f2450199f598dd02ad6"
server_key_iv: "60e0d8153764eda7e9cdf6b5b56fd7a2"
server_key_version: "encryptionkey@0"
I20260925 21:59:34.423394 18786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:34.441375 18786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:34.444036 18786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:34.444195 18786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:34.444475 18786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:34.444981 18839 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 21:59:34.446458 18786 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 21:59:34.446519 18786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 21:59:34.446553 18786 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 21:59:34.447414 18786 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 21:59:34.447466 18786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 21:59:34.447521 18839 tablet_bootstrap.cc:492] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: Bootstrap starting.
I20260925 21:59:34.449184 18839 log.cc:824] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:34.452090 18839 tablet_bootstrap.cc:492] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 21:59:34.452385 18839 tablet_bootstrap.cc:492] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: Bootstrap complete.
I20260925 21:59:34.452539 18839 ts_tablet_manager.cc:1397] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260925 21:59:34.454308 18786 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:33061
I20260925 21:59:34.454411 18947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:33061 every 8 connection(s)
I20260925 21:59:34.454185 18839 raft_consensus.cc:348] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:34.454555 18839 raft_consensus.cc:729] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e43ab7079304b4999f0f6927d2b3c0d, State: Initialized, Role: FOLLOWER
I20260925 21:59:34.454701 18786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:59:34.454723 18839 consensus_queue.cc:260] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:34.455057 18839 ts_tablet_manager.cc:1428] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20260925 21:59:34.458858 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 18786
I20260925 21:59:34.459151 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:35691
--local_ip_for_outbound_sockets=127.2.70.194
--tserver_master_addrs=127.2.70.254:32829
--webserver_port=45607
--webserver_interface=127.2.70.194
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:34.508725 18951 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 21:59:34.455372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:57671)
0925 21:59:34.455514 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:34.455564 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:34.455694 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:34.508522 (+ 52828us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
W20260925 21:59:34.508982 18948 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:32829 (0 consecutive failures): Network error: Failed to ping master at 127.2.70.254:32829: Client connection negotiation failed: client connection to 127.2.70.254:32829: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 21:59:34.509320 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:34.509192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:43695)
0925 21:59:34.509241 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:34.509280 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 21:59:34.509682 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:34.509597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:42083)
0925 21:59:34.509637 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:34.509662 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 21:59:34.509832 18948 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 21:59:34.539914 18952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:34.540129 18952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:34.540153 18952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:34.541361 18952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:34.541407 18952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:34.541433 18952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:59:34.542913 18952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:35691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=45607
--tserver_master_addrs=127.2.70.254:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:34.543154 18952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:34.543366 18952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:34.548363 18960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:34.548384 18957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:34.548384 18952 server_base.cc:1034] running on GCE node
W20260925 21:59:34.548522 18958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:34.548779 18952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:34.549000 18952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:34.550144 18952 hybrid_clock.cc:648] HybridClock initialized: now 1790373574550130 us; error 27 us; skew 500 ppm
Sep 25 21:59:34 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373574, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:34.552479 18952 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:59:34.552883 18952 webserver.cc:460] Webserver started at http://127.2.70.194:45607/ using document root <none> and password file <none>
I20260925 21:59:34.553088 18952 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:34.553128 18952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:34.554349 18952 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:34.554914 18967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:34.555126 18952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:59:34.555205 18952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "1d1d335cbcbc459ca9eab6995926b1d8"
format_stamp: "Formatted at 2026-09-25 21:58:52 on dist-test-slave-2g0f"
server_key: "ebd36810ec28cec268ac367f87f362e4"
server_key_iv: "1f8ef791e75cc9e81df39baa9a8e274d"
server_key_version: "encryptionkey@0"
I20260925 21:59:34.555532 18952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:34.574744 18952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:34.576977 18952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:34.577126 18952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:34.577347 18952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:34.577641 18952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:34.577673 18952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:34.577703 18952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:34.577728 18952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:34.583392 18952 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:35691
I20260925 21:59:34.583520 19080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:35691 every 8 connection(s)
I20260925 21:59:34.583794 18952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:59:34.584012 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 18952
I20260925 21:59:34.584311 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:43127
--local_ip_for_outbound_sockets=127.2.70.195
--tserver_master_addrs=127.2.70.254:32829
--webserver_port=38087
--webserver_interface=127.2.70.195
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 21:59:34.585037 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:34.584550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:46013)
0925 21:59:34.584723 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:34.584822 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
W20260925 21:59:34.585440 19081 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:32829 (0 consecutive failures): Network error: Failed to ping master at 127.2.70.254:32829: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
I20260925 21:59:34.585711 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:34.585617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:43909)
0925 21:59:34.585660 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:34.585686 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 21:59:34.586011 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:34.585925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:51715)
0925 21:59:34.585962 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:34.585983 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 21:59:34.586139 19081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 21:59:34.670825 19084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:34.671026 19084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:34.671048 19084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:34.672224 19084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:34.672264 19084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:34.672290 19084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:59:34.673806 19084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:43127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=38087
--tserver_master_addrs=127.2.70.254:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:34.674005 19084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:34.674177 19084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:34.679169 19092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:34.679203 19090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:34.679200 19089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:34.679349 19084 server_base.cc:1034] running on GCE node
I20260925 21:59:34.679584 19084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:34.679793 19084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:34.680953 19084 hybrid_clock.cc:648] HybridClock initialized: now 1790373574680936 us; error 29 us; skew 500 ppm
Sep 25 21:59:34 dist-test-slave-2g0f krb5kdc[17644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373574, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:34.683352 19084 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:59:34.683768 19084 webserver.cc:460] Webserver started at http://127.2.70.195:38087/ using document root <none> and password file <none>
I20260925 21:59:34.684006 19084 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:34.684055 19084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:34.685372 19084 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:34.685961 19099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:34.686116 19084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:34.686208 19084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "4e39fa99a752489aa942336750eddef4"
format_stamp: "Formatted at 2026-09-25 21:58:53 on dist-test-slave-2g0f"
server_key: "109849c09d76632e0e541e251f3846fb"
server_key_iv: "ca9a44b13f6c9f079636c7631edb8058"
server_key_version: "encryptionkey@0"
I20260925 21:59:34.686556 19084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:34.702793 19084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:34.704873 19084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:34.705015 19084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:34.705240 19084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:34.705547 19084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:34.705581 19084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:34.705611 19084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:34.705633 19084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:34.711061 19084 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:43127
I20260925 21:59:34.711117 19212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:43127 every 8 connection(s)
I20260925 21:59:34.711439 19084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:59:34.712131 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 19084
I20260925 21:59:34.712527 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:34.712083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:36559)
0925 21:59:34.712247 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:34.712327 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
W20260925 21:59:34.712893 19213 heartbeater.cc:644] Failed to heartbeat to 127.2.70.254:32829 (0 consecutive failures): Network error: Failed to ping master at 127.2.70.254:32829: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
I20260925 21:59:34.713143 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:34.713049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:50809)
0925 21:59:34.713090 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:34.713122 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 21:59:34.713374 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:34.713302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:54155)
0925 21:59:34.713334 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:34.713354 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 21:59:34.713460 19213 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 21:59:35.510632 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:35.510324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:40185)
0925 21:59:35.510520 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:35.510568 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260925 21:59:35.586606 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:35.586416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:56011)
0925 21:59:35.586522 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:35.586552 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260925 21:59:35.713968 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:35.713732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:48437)
0925 21:59:35.713869 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:35.713910 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":54,"threads_started":1}
I20260925 21:59:36.150981 19220 raft_consensus.cc:482] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:59:36.151117 19220 raft_consensus.cc:504] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:36.151432 19220 leader_election.cc:288] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691), 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127)
Sep 25 21:59:36 dist-test-slave-2g0f krb5kdc[17644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373574, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.195@KRBTEST.COM
Sep 25 21:59:36 dist-test-slave-2g0f krb5kdc[17644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373574, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.194@KRBTEST.COM
I20260925 21:59:36.162814 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:36.151799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:43127 (local address 127.2.70.193:54167)
0925 21:59:36.151993 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:36.152001 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:36.152038 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:36.152724 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:36.152733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:36.153129 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:36.153358 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:36.153364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:36.154523 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:36.154524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:36.155377 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:36.155379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:36.156076 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:36.156077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:36.156206 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:36.156777 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:36.156781 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:36.158240 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:36.160586 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:36.160596 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:36.160598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:36.161072 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:36.161392 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:36.161393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:36.161393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:36.161568 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:36.162168 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:36.162176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:36.162350 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 21:59:36.162474 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 21:59:36.162662 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":81,"thread_start_us":77,"threads_started":1}
I20260925 21:59:36.162833 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:36.151759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:56227 (local address 127.2.70.194:35691)
0925 21:59:36.151977 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:36.151989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:36.152032 (+ 43us) server_negotiation.cc:407] Connection header received
0925 21:59:36.152488 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:36.152490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:36.152532 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:36.152653 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:36.153632 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:36.154522 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:36.155467 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:36.155981 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:36.156089 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:36.158681 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:36.158686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:36.158686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:36.158699 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:36.160499 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:36.161122 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:36.161125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:36.161126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:36.161278 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:36.161667 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:36.161669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:36.161669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:36.162127 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:36.162194 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:36.162572 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 21:59:36.162719 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260925 21:59:36.162833 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:36.151941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:54167 (local address 127.2.70.195:43127)
0925 21:59:36.152085 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:36.152095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:36.152135 (+ 40us) server_negotiation.cc:407] Connection header received
0925 21:59:36.152545 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:36.152547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:36.152583 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:36.152727 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:36.153500 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:36.154424 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:36.155470 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:36.155997 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:36.156103 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:36.158365 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:36.158370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:36.158370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:36.158385 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:36.160450 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:36.161160 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:36.161162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:36.161163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:36.161277 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:36.161668 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:36.161669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:36.161669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:36.162035 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:36.162130 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:36.162572 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 21:59:36.162719 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260925 21:59:36.162825 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:36.151639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:35691 (local address 127.2.70.193:56227)
0925 21:59:36.151764 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:36.151779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:36.151897 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:36.152724 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:36.152727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:36.153128 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:36.153518 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:36.153521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:36.154616 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:36.154617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:36.155379 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:36.155381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:36.156072 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:36.156073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:36.156205 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:36.156725 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:36.156733 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:36.158586 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:36.160586 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:36.160596 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:36.160598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:36.161011 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:36.161388 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:36.161389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:36.161390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:36.161570 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:36.162208 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:36.162209 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:36.162350 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 21:59:36.162475 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 21:59:36.162730 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":146,"thread_start_us":65,"threads_started":1}
W20260925 21:59:36.164160 18833 leader_election.cc:341] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691): Not found: Tablet not found: f0a6dbbda52f4932804722cfd487f6e4
W20260925 21:59:36.164237 18833 leader_election.cc:341] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127): Not found: Tablet not found: f0a6dbbda52f4932804722cfd487f6e4
I20260925 21:59:36.164253 18833 leader_election.cc:302] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e43ab7079304b4999f0f6927d2b3c0d; no voters: 1d1d335cbcbc459ca9eab6995926b1d8, 4e39fa99a752489aa942336750eddef4
I20260925 21:59:36.164354 19220 raft_consensus.cc:2726] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 21:59:36.511420 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:36.511231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:43589)
0925 21:59:36.511316 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:36.511365 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 21:59:36.587535 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:36.587257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:60397)
0925 21:59:36.587421 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:36.587484 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260925 21:59:36.714640 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:36.714397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:56755)
0925 21:59:36.714553 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:36.714598 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":58,"threads_started":1}
I20260925 21:59:37.512214 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:37.511917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:56341)
0925 21:59:37.512108 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:37.512156 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260925 21:59:37.588325 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:37.588098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:59895)
0925 21:59:37.588235 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:37.588279 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260925 21:59:37.681032 19229 raft_consensus.cc:482] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:59:37.681128 19229 raft_consensus.cc:504] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:37.681257 19229 leader_election.cc:288] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691), 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127)
W20260925 21:59:37.681746 18833 leader_election.cc:341] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691): Not found: Tablet not found: f0a6dbbda52f4932804722cfd487f6e4
W20260925 21:59:37.681808 18833 leader_election.cc:341] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127): Not found: Tablet not found: f0a6dbbda52f4932804722cfd487f6e4
I20260925 21:59:37.681824 18833 leader_election.cc:302] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e43ab7079304b4999f0f6927d2b3c0d; no voters: 1d1d335cbcbc459ca9eab6995926b1d8, 4e39fa99a752489aa942336750eddef4
I20260925 21:59:37.681891 19229 raft_consensus.cc:2726] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 21:59:37.715332 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:37.715083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:46985)
0925 21:59:37.715221 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:37.715273 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260925 21:59:38.512954 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:38.512733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:53537)
0925 21:59:38.512859 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:38.512895 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260925 21:59:38.589129 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:38.588883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:57947)
0925 21:59:38.589042 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:38.589084 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":61,"threads_started":1}
I20260925 21:59:38.716078 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:38.715811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:59287)
0925 21:59:38.715987 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:38.716031 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":105,"threads_started":1}
I20260925 21:59:39.513676 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:39.513401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:39759)
0925 21:59:39.513586 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:39.513628 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260925 21:59:39.589887 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:39.589652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:34827)
0925 21:59:39.589791 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:39.589841 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260925 21:59:39.716724 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:39.716491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:56599)
0925 21:59:39.716639 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:39.716678 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260925 21:59:39.955865 19237 raft_consensus.cc:482] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:59:39.955973 19237 raft_consensus.cc:504] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:39.956135 19237 leader_election.cc:288] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691), 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127)
W20260925 21:59:39.956498 18833 leader_election.cc:341] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691): Not found: Tablet not found: f0a6dbbda52f4932804722cfd487f6e4
W20260925 21:59:39.956564 18833 leader_election.cc:341] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127): Not found: Tablet not found: f0a6dbbda52f4932804722cfd487f6e4
I20260925 21:59:39.956589 18833 leader_election.cc:302] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e43ab7079304b4999f0f6927d2b3c0d; no voters: 1d1d335cbcbc459ca9eab6995926b1d8, 4e39fa99a752489aa942336750eddef4
I20260925 21:59:39.956676 19237 raft_consensus.cc:2726] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 21:59:40.514456 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:40.514219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:55505)
0925 21:59:40.514347 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:40.514408 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260925 21:59:40.590667 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:40.590408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:44761)
0925 21:59:40.590560 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:40.590603 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260925 21:59:40.717432 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:40.717218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:36057)
0925 21:59:40.717354 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:40.717385 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260925 21:59:41.515239 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:41.514988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:43547)
0925 21:59:41.515144 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:41.515190 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260925 21:59:41.591365 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:41.591179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:37469)
0925 21:59:41.591288 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:41.591322 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260925 21:59:41.718176 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:41.717890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:42711)
0925 21:59:41.718078 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:41.718115 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":111,"threads_started":1}
I20260925 21:59:41.959672 19244 raft_consensus.cc:482] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:59:41.959777 19244 raft_consensus.cc:504] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:41.959986 19244 leader_election.cc:288] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691), 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127)
W20260925 21:59:41.960323 18833 leader_election.cc:341] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691): Not found: Tablet not found: f0a6dbbda52f4932804722cfd487f6e4
W20260925 21:59:41.960395 18833 leader_election.cc:341] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127): Not found: Tablet not found: f0a6dbbda52f4932804722cfd487f6e4
I20260925 21:59:41.960412 18833 leader_election.cc:302] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e43ab7079304b4999f0f6927d2b3c0d; no voters: 1d1d335cbcbc459ca9eab6995926b1d8, 4e39fa99a752489aa942336750eddef4
I20260925 21:59:41.960475 19244 raft_consensus.cc:2726] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 21:59:42.516058 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:42.515812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:39831)
0925 21:59:42.515969 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:42.516013 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260925 21:59:42.592105 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:42.591729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:56309)
0925 21:59:42.592022 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:42.592059 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":60,"threads_started":1}
I20260925 21:59:42.719048 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:42.718758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:37501)
0925 21:59:42.718884 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:42.718928 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260925 21:59:43.516721 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:43.516499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:49041)
0925 21:59:43.516633 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:43.516672 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260925 21:59:43.592756 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:43.592559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:43003)
0925 21:59:43.592668 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:43.592705 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260925 21:59:43.719731 19250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:43.719497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:47791)
0925 21:59:43.719644 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:43.719684 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260925 21:59:44.517477 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:44.517268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:52343)
0925 21:59:44.517389 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:44.517430 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260925 21:59:44.593461 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:44.593235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:42913)
0925 21:59:44.593367 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:44.593407 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260925 21:59:44.720408 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:44.720173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:41117)
0925 21:59:44.720315 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:44.720360 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260925 21:59:45.518194 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:45.517940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:32999)
0925 21:59:45.518115 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:45.518147 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":116,"threads_started":1}
I20260925 21:59:45.594091 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:45.593878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:57995)
0925 21:59:45.594011 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:45.594045 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260925 21:59:45.721123 19256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:45.720856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:54635)
0925 21:59:45.721023 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:45.721070 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260925 21:59:46.518874 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:46.518679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:44655)
0925 21:59:46.518793 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:46.518822 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260925 21:59:46.594759 19258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:46.594518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:58395)
0925 21:59:46.594664 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:46.594707 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260925 21:59:46.721807 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:46.721555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:50791)
0925 21:59:46.721719 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:46.721760 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260925 21:59:46.840538 19260 raft_consensus.cc:482] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:59:46.840633 19260 raft_consensus.cc:504] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d335cbcbc459ca9eab6995926b1d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 35691 } } peers { permanent_uuid: "4e39fa99a752489aa942336750eddef4" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 43127 } } peers { permanent_uuid: "6e43ab7079304b4999f0f6927d2b3c0d" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 33061 } }
I20260925 21:59:46.840767 19260 leader_election.cc:288] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691), 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127)
W20260925 21:59:46.841176 18833 leader_election.cc:341] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e39fa99a752489aa942336750eddef4 (127.2.70.195:43127): Not found: Tablet not found: f0a6dbbda52f4932804722cfd487f6e4
W20260925 21:59:46.841240 18833 leader_election.cc:341] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d1d335cbcbc459ca9eab6995926b1d8 (127.2.70.194:35691): Not found: Tablet not found: f0a6dbbda52f4932804722cfd487f6e4
I20260925 21:59:46.841261 18833 leader_election.cc:302] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e43ab7079304b4999f0f6927d2b3c0d; no voters: 1d1d335cbcbc459ca9eab6995926b1d8, 4e39fa99a752489aa942336750eddef4
I20260925 21:59:46.841324 19260 raft_consensus.cc:2726] T f0a6dbbda52f4932804722cfd487f6e4 P 6e43ab7079304b4999f0f6927d2b3c0d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 21:59:47.519608 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:47.519354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:49529)
0925 21:59:47.519508 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:47.519547 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260925 21:59:47.595494 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:47.595265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:46807)
0925 21:59:47.595406 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:47.595448 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260925 21:59:47.722604 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:47.722306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:55387)
0925 21:59:47.722473 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:47.722515 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
I20260925 21:59:48.520287 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:48.520063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:34247)
0925 21:59:48.520199 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:48.520237 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260925 21:59:48.596248 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:48.596001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:48525)
0925 21:59:48.596155 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:48.596200 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":62,"threads_started":1}
I20260925 21:59:48.723318 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:48.723103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:37417)
0925 21:59:48.723226 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:48.723269 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260925 21:59:49.521028 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:49.520831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.193:46465)
0925 21:59:49.520945 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:49.520983 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260925 21:59:49.597023 19268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:49.596803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.194:42437)
0925 21:59:49.596933 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:49.596976 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260925 21:59:49.717288 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 18786
I20260925 21:59:49.721491 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 18952
I20260925 21:59:49.724010 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:49.723804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:32829 (local address 127.2.70.195:51741)
0925 21:59:49.723931 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:49.723962 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.254:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260925 21:59:49.725289 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 19084
2026-09-25T21:59:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58252 ms)
[----------] 30 tests from SecurityITest (232321 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:59:49 dist-test-slave-2g0f krb5kdc[19272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:59:49 dist-test-slave-2g0f krb5kdc[19272](info): set up 2 sockets
Sep 25 21:59:49 dist-test-slave-2g0f krb5kdc[19272](info): commencing operation
krb5kdc: starting...
W20260925 21:59:50.687374 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:59:50 dist-test-slave-2g0f krb5kdc[19272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:59:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:59:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:59:50.807969 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:43019
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:50.887720 19285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:50.887984 19285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:50.888020 19285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:50.889261 19285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:59:50.889310 19285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:50.889329 19285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:50.889345 19285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:59:50.889359 19285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:59:50.890965 19285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:43019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:50.891232 19285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:50.891453 19285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 21:59:50.896721 19285 server_base.cc:1034] running on GCE node
W20260925 21:59:50.896827 19293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:50.896894 19290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:50.896839 19291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:50.897253 19285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:50.897553 19285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:50.898708 19285 hybrid_clock.cc:648] HybridClock initialized: now 1790373590898696 us; error 28 us; skew 500 ppm
Sep 25 21:59:50 dist-test-slave-2g0f krb5kdc[19272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373590, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:50.905015 19285 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:59:50.905503 19285 webserver.cc:460] Webserver started at http://127.2.70.254:39411/ using document root <none> and password file <none>
I20260925 21:59:50.905730 19285 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:50.905779 19285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:50.905905 19285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:50.906955 19285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "43158a616ca640f38469676004a890b2"
format_stamp: "Formatted at 2026-09-25 21:59:50 on dist-test-slave-2g0f"
server_key: "a82c82802d64c17aac4fef12ec14f7f9"
server_key_iv: "cbb917745ee3a82ba7cea81a9f3e465f"
server_key_version: "encryptionkey@0"
I20260925 21:59:50.907279 19285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "43158a616ca640f38469676004a890b2"
format_stamp: "Formatted at 2026-09-25 21:59:50 on dist-test-slave-2g0f"
server_key: "a82c82802d64c17aac4fef12ec14f7f9"
server_key_iv: "cbb917745ee3a82ba7cea81a9f3e465f"
server_key_version: "encryptionkey@0"
I20260925 21:59:50.908670 19285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:59:50.909483 19300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:50.909703 19285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:50.909791 19285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "43158a616ca640f38469676004a890b2"
format_stamp: "Formatted at 2026-09-25 21:59:50 on dist-test-slave-2g0f"
server_key: "a82c82802d64c17aac4fef12ec14f7f9"
server_key_iv: "cbb917745ee3a82ba7cea81a9f3e465f"
server_key_version: "encryptionkey@0"
I20260925 21:59:50.909853 19285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:50.931654 19285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:50.933830 19285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:50.933975 19285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:50.937922 19285 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:43019
I20260925 21:59:50.937951 19352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:43019 every 8 connection(s)
I20260925 21:59:50.938284 19285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:59:50.939127 19353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:50.941429 19353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2: Bootstrap starting.
I20260925 21:59:50.941979 19353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:50.942245 19353 log.cc:824] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:50.942910 19353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2: No bootstrap required, opened a new log
I20260925 21:59:50.942970 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 19285
I20260925 21:59:50.943063 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:59:50.943238 2331 external_mini_cluster.cc:1428] Setting key 8206a8aa074eeb508665c538c63eddd3
I20260925 21:59:50.945195 19353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43158a616ca640f38469676004a890b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43019 } }
I20260925 21:59:50.945367 19353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:50.945412 19353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43158a616ca640f38469676004a890b2, State: Initialized, Role: FOLLOWER
I20260925 21:59:50.945544 19353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43158a616ca640f38469676004a890b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43019 } }
I20260925 21:59:50.945618 19353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:59:50.945654 19353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:59:50.945698 19353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:50.946358 19353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43158a616ca640f38469676004a890b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43019 } }
I20260925 21:59:50.946463 19353 leader_election.cc:302] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43158a616ca640f38469676004a890b2; no voters:
I20260925 21:59:50.946610 19353 leader_election.cc:288] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:59:50.946643 19358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:50.946801 19358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [term 1 LEADER]: Becoming Leader. State: Replica: 43158a616ca640f38469676004a890b2, State: Running, Role: LEADER
I20260925 21:59:50.946841 19353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:59:50.946903 19358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43158a616ca640f38469676004a890b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43019 } }
I20260925 21:59:50.947216 19358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43158a616ca640f38469676004a890b2. Latest consensus state: current_term: 1 leader_uuid: "43158a616ca640f38469676004a890b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43158a616ca640f38469676004a890b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43019 } } }
I20260925 21:59:50.947243 19359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43158a616ca640f38469676004a890b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43158a616ca640f38469676004a890b2" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43019 } } }
I20260925 21:59:50.947418 19359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:50.947608 19358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2 [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:50.948035 19367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:59:50.948609 19367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 21:59:50 dist-test-slave-2g0f krb5kdc[19272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:50.950940 19367 catalog_manager.cc:1347] Generated new cluster ID: 272fd35a58aa4998bc11047190416fc8
I20260925 21:59:50.950989 19367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:59:50.954123 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:50.943713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.2.70.254:43019)
0925 21:59:50.943870 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:50.943879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:50.943937 (+ 58us) server_negotiation.cc:407] Connection header received
0925 21:59:50.944378 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:50.944388 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:50.944457 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:50.944606 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:50.945150 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:50.946811 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:50.947955 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:50.948505 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:50.948612 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:50.951254 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:50.951258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:50.951258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:50.951268 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:50.952818 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:50.953127 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:50.953130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:50.953131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:50.953209 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:50.953372 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:50.953373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:50.953373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:50.953596 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:50.953742 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:50.953819 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 21:59:50.953947 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260925 21:59:50.959573 19367 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:59:50.960261 19367 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:59:50.964489 19367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2: Generated new TSK 0
I20260925 21:59:50.964668 19367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:59:50.995132 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:43019
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:51.098758 19379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:51.098965 19379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:51.098989 19379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:51.100268 19379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:51.100310 19379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:51.100337 19379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:59:51.101812 19379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:51.102041 19379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:51.102270 19379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:51.107220 19387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:51.107352 19385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:51.107458 19384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:51.107501 19379 server_base.cc:1034] running on GCE node
I20260925 21:59:51.107752 19379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:51.108013 19379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:51.109189 19379 hybrid_clock.cc:648] HybridClock initialized: now 1790373591109172 us; error 30 us; skew 500 ppm
Sep 25 21:59:51 dist-test-slave-2g0f krb5kdc[19272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373591, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:51.112049 19379 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:59:51.112551 19379 webserver.cc:460] Webserver started at http://127.2.70.193:34837/ using document root <none> and password file <none>
I20260925 21:59:51.112810 19379 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:51.112869 19379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:51.113016 19379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:51.114223 19379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "47a1633350d04fffa6c77cb103f40681"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "12a4768b4065ba06c9f0b585e17ca9a0"
server_key_iv: "be8b47525b16e24a571003c67fff764c"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.114589 19379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "47a1633350d04fffa6c77cb103f40681"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "12a4768b4065ba06c9f0b585e17ca9a0"
server_key_iv: "be8b47525b16e24a571003c67fff764c"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.116226 19379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 21:59:51.116936 19394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.117165 19379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:51.117255 19379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "47a1633350d04fffa6c77cb103f40681"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "12a4768b4065ba06c9f0b585e17ca9a0"
server_key_iv: "be8b47525b16e24a571003c67fff764c"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.117326 19379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:51.149120 19379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:51.151309 19379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:51.151438 19379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:51.151669 19379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:51.152007 19379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:51.152042 19379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.152064 19379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:51.152154 19379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.157452 19379 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:45085
I20260925 21:59:51.157495 19507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:45085 every 8 connection(s)
I20260925 21:59:51.157824 19379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:59:51.161139 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 19379
I20260925 21:59:51.161248 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:59:51.161429 2331 external_mini_cluster.cc:1428] Setting key 388e5ca16a4f902ce3da9fafcb56838a
Sep 25 21:59:51 dist-test-slave-2g0f krb5kdc[19272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373591, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:51.167672 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.158530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:58151 (local address 127.2.70.254:43019)
0925 21:59:51.158592 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:51.158594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:51.158785 (+ 191us) server_negotiation.cc:407] Connection header received
0925 21:59:51.159350 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:51.159352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:51.159364 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:51.159416 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:51.160388 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.160897 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.161947 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.162164 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.162220 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.164643 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:51.164645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:51.164645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:51.164653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:51.166009 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:51.166531 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:51.166533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:51.166533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:51.166598 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:51.166913 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:51.166914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:51.166915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:51.167057 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:51.167117 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:51.167490 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 21:59:51.167599 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 21:59:51.167894 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.158498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43019 (local address 127.2.70.193:58151)
0925 21:59:51.158661 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:51.158698 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:51.159242 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:51.159486 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:51.159490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:51.160094 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:51.160292 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.160297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.160984 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.160985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.161863 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.161866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.162235 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.162236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.162353 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.162905 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:51.162921 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:51.164525 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:51.166105 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:51.166115 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:51.166118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:51.166432 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:51.166673 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:51.166674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:51.166674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:51.166817 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:51.167124 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:51.167130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:51.167313 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 21:59:51.167435 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 21:59:51.167625 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260925 21:59:51.168426 19508 heartbeater.cc:347] Connected to a master server at 127.2.70.254:43019
I20260925 21:59:51.168527 19508 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:51.168720 19508 heartbeater.cc:511] Master 127.2.70.254:43019 requested a full tablet report, sending...
I20260925 21:59:51.169312 19317 ts_manager.cc:194] Registered new tserver with Master: 47a1633350d04fffa6c77cb103f40681 (127.2.70.193:45085)
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
I20260925 21:59:51.170184 19317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:58151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:59:51.188951 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:43019
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:51.269564 19513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:51.269810 19513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:51.269834 19513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:51.271075 19513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:51.271126 19513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:51.271152 19513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:59:51.272706 19513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:51.272943 19513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:51.273173 19513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:51.278326 19521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:51.278394 19513 server_base.cc:1034] running on GCE node
W20260925 21:59:51.278350 19518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:51.278343 19519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:51.278725 19513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:51.278963 19513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:51.280112 19513 hybrid_clock.cc:648] HybridClock initialized: now 1790373591280095 us; error 28 us; skew 500 ppm
Sep 25 21:59:51 dist-test-slave-2g0f krb5kdc[19272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373591, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:51.282545 19513 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:59:51.282960 19513 webserver.cc:460] Webserver started at http://127.2.70.194:35271/ using document root <none> and password file <none>
I20260925 21:59:51.283154 19513 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:51.283201 19513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:51.283325 19513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:51.284394 19513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "af73652e01f74bc4b4080b66317e8aa8"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "570621bb5f95508e69dba7e08c9ad493"
server_key_iv: "2a550a3aed3299bb295460852ed2e76a"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.284726 19513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "af73652e01f74bc4b4080b66317e8aa8"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "570621bb5f95508e69dba7e08c9ad493"
server_key_iv: "2a550a3aed3299bb295460852ed2e76a"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.285902 19513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:59:51.286586 19528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.286741 19513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:51.286818 19513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "af73652e01f74bc4b4080b66317e8aa8"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "570621bb5f95508e69dba7e08c9ad493"
server_key_iv: "2a550a3aed3299bb295460852ed2e76a"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.286875 19513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:51.305362 19513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:51.307574 19513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:51.307711 19513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:51.307969 19513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:51.308313 19513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:51.308357 19513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.308393 19513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:51.308410 19513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.314095 19513 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:33509
I20260925 21:59:51.314149 19641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:33509 every 8 connection(s)
I20260925 21:59:51.314459 19513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
Sep 25 21:59:51 dist-test-slave-2g0f krb5kdc[19272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373591, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:51.323547 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.315114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:35779 (local address 127.2.70.254:43019)
0925 21:59:51.315182 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:51.315184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:51.315343 (+ 159us) server_negotiation.cc:407] Connection header received
0925 21:59:51.315966 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:51.315968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:51.315980 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:51.316047 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:51.316997 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.317500 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.318493 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.318698 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.318754 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.320982 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:51.320984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:51.320985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:51.320995 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:51.322351 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:51.322741 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:51.322742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:51.322742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:51.322795 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:51.322981 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:51.322982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:51.322982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:51.323088 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:51.323144 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:51.323397 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 21:59:51.323480 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 21:59:51.323695 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.315060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43019 (local address 127.2.70.194:35779)
0925 21:59:51.315216 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:51.315253 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:51.315744 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:51.316128 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:51.316132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:51.316707 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:51.316908 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.316912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.317596 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.317597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.318413 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.318416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.318779 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.318780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.318874 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.319333 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:51.319342 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:51.320824 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:51.322429 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:51.322435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:51.322437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:51.322670 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:51.322847 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:51.322847 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:51.322848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:51.322936 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:51.323139 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:51.323142 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:51.323258 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 21:59:51.323334 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 21:59:51.323467 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260925 21:59:51.324225 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 19513
I20260925 21:59:51.324234 19642 heartbeater.cc:347] Connected to a master server at 127.2.70.254:43019
I20260925 21:59:51.324326 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:59:51.324329 19642 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:51.324545 19642 heartbeater.cc:511] Master 127.2.70.254:43019 requested a full tablet report, sending...
I20260925 21:59:51.324550 2331 external_mini_cluster.cc:1428] Setting key 7d2c0b9175bf7aa443f18dcaa6b0feb9
I20260925 21:59:51.324934 19316 ts_manager.cc:194] Registered new tserver with Master: af73652e01f74bc4b4080b66317e8aa8 (127.2.70.194:33509)
I20260925 21:59:51.325394 19316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:35779
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:59:51.350364 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:43019
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:51.430835 19647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:51.431075 19647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:51.431111 19647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:51.432439 19647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:51.432497 19647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:51.432536 19647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:59:51.434134 19647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:51.434410 19647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:51.434643 19647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:51.439697 19655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:51.439806 19652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:51.439746 19653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:51.440043 19647 server_base.cc:1034] running on GCE node
I20260925 21:59:51.440259 19647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:51.440557 19647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:51.441740 19647 hybrid_clock.cc:648] HybridClock initialized: now 1790373591441719 us; error 31 us; skew 500 ppm
Sep 25 21:59:51 dist-test-slave-2g0f krb5kdc[19272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373591, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:51.444434 19647 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:59:51.444905 19647 webserver.cc:460] Webserver started at http://127.2.70.195:37301/ using document root <none> and password file <none>
I20260925 21:59:51.445119 19647 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:51.445168 19647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:51.445295 19647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:51.446404 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "1495070baff9438faebb3677f5c7ebb6"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "2dbd81ef451cefc6eb61b88330e17b2c"
server_key_iv: "8502eb8763e7b3fac5474860df32af00"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.446754 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "1495070baff9438faebb3677f5c7ebb6"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "2dbd81ef451cefc6eb61b88330e17b2c"
server_key_iv: "8502eb8763e7b3fac5474860df32af00"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.448307 19647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:59:51.449023 19662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.449201 19647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:51.449285 19647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "1495070baff9438faebb3677f5c7ebb6"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "2dbd81ef451cefc6eb61b88330e17b2c"
server_key_iv: "8502eb8763e7b3fac5474860df32af00"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.449354 19647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:51.458142 19647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:51.460633 19647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:51.460788 19647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:51.461025 19647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:51.461331 19647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:51.461369 19647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.461401 19647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:51.461429 19647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.467875 19647 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:39585
I20260925 21:59:51.467916 19775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:39585 every 8 connection(s)
I20260925 21:59:51.468262 19647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
Sep 25 21:59:51 dist-test-slave-2g0f krb5kdc[19272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373591, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:51.475143 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 19647
I20260925 21:59:51.475246 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:59:51.475412 2331 external_mini_cluster.cc:1428] Setting key 0797abc56f36c5ecc14b92a91acb5106
I20260925 21:59:51.477481 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.468993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:43373 (local address 127.2.70.254:43019)
0925 21:59:51.469074 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:51.469076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:51.469241 (+ 165us) server_negotiation.cc:407] Connection header received
0925 21:59:51.469755 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:51.469756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:51.469769 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:51.469830 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:51.470788 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.471315 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.472362 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.472581 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.472645 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.474692 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:51.474694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:51.474694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:51.474704 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:51.476248 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:51.476611 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:51.476612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:51.476613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:51.476672 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:51.476883 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:51.476884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:51.476884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:51.477002 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:51.477056 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:51.477312 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 21:59:51.477391 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 21:59:51.477662 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.468974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:43019 (local address 127.2.70.195:43373)
0925 21:59:51.469121 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:51.469156 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:51.469637 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:51.469892 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:51.469896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:51.470460 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:51.470686 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.470692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.471395 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.471396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.472277 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.472280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.472647 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.472648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.472741 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.473206 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:51.473215 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:51.474586 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:51.476318 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:51.476325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:51.476327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:51.476542 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:51.476729 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:51.476730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:51.476730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:51.476830 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:51.477055 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:51.477062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:51.477199 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 21:59:51.477286 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 21:59:51.477449 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260925 21:59:51.478116 19776 heartbeater.cc:347] Connected to a master server at 127.2.70.254:43019
I20260925 21:59:51.478216 19776 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:51.478395 19776 heartbeater.cc:511] Master 127.2.70.254:43019 requested a full tablet report, sending...
I20260925 21:59:51.478794 19316 ts_manager.cc:194] Registered new tserver with Master: 1495070baff9438faebb3677f5c7ebb6 (127.2.70.195:39585)
I20260925 21:59:51.479167 19316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:43373
I20260925 21:59:51.487470 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:59:51.494809 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.488872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.2.70.254:43019)
0925 21:59:51.488949 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:51.488952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:51.489054 (+ 102us) server_negotiation.cc:407] Connection header received
0925 21:59:51.489118 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:51.489119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:51.489129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:51.489191 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:51.489762 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.490295 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.491047 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.491238 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.491291 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.492457 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:51.492458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:51.492459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:51.492475 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:51.493721 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:51.494039 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:51.494041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:51.494041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:51.494093 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:51.494253 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:51.494254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:51.494254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:51.494410 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:51.494511 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:51.494633 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 21:59:51.494718 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 21:59:51.496742 19316 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:59:51.498314 19316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:59:51.504633 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.500699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:39585 (local address 127.0.0.1:45442)
0925 21:59:51.500959 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:51.500968 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:51.501025 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:51.501353 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:51.501353 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:51.501355 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:51.501390 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.501391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.502268 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.502269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.503552 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.503555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.504337 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.504338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.504418 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.504432 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 21:59:51.504498 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 21:59:51.504552 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":30,"thread_start_us":71,"threads_started":1}
I20260925 21:59:51.504787 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.500966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.2.70.193:45085)
0925 21:59:51.501112 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:51.501115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:51.501238 (+ 123us) server_negotiation.cc:407] Connection header received
0925 21:59:51.501278 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:51.501280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:51.501294 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:51.501361 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:51.501534 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.502589 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.503598 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.504317 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.504382 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.504437 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:51.504674 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 21:59:51.504712 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260925 21:59:51.504905 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.500618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.2.70.194:33509)
0925 21:59:51.501003 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:51.501008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:51.501017 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:59:51.501057 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:51.501059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:51.501116 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:51.501193 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:51.501373 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.502474 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.503693 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.504405 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.504466 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.504515 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:51.504595 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 21:59:51.504634 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":96,"threads_started":1}
I20260925 21:59:51.505187 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.500808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.2.70.195:39585)
0925 21:59:51.500980 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:51.500985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:51.501014 (+ 29us) server_negotiation.cc:407] Connection header received
0925 21:59:51.501111 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:51.501114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:51.501128 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:51.501195 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:51.501462 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.502178 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.503657 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.504261 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.504326 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.504387 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:51.505082 (+ 695us) server_negotiation.cc:299] Negotiation successful
0925 21:59:51.505126 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260925 21:59:51.504721 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.500501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:33509 (local address 127.0.0.1:51402)
0925 21:59:51.500629 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:51.500672 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:51.500740 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:51.501212 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:51.501215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:51.501217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:51.501289 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.501291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.502569 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.502570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.503626 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.503628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.504474 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.504475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.504533 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.504536 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:51.504581 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 21:59:51.504627 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260925 21:59:51.504801 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.500855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:45085 (local address 127.0.0.1:60712)
0925 21:59:51.501130 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:51.501139 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:51.501189 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:51.501442 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:51.501443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:51.501445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:51.501473 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.501474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.502673 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.502674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.503521 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.503524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.504471 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.504472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.504535 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.504540 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:59:51.504592 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 21:59:51.504636 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":1,"thread_start_us":55,"threads_started":1}
I20260925 21:59:51.505956 19576 tablet_service.cc:1467] Processing CreateTablet for tablet e06eefd42819411495f9ad7e328fd58d (DEFAULT_TABLE table=test-table [id=b165ac61b3574ee3984395cb82aef241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:51.505956 19710 tablet_service.cc:1467] Processing CreateTablet for tablet e06eefd42819411495f9ad7e328fd58d (DEFAULT_TABLE table=test-table [id=b165ac61b3574ee3984395cb82aef241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:51.505992 19442 tablet_service.cc:1467] Processing CreateTablet for tablet e06eefd42819411495f9ad7e328fd58d (DEFAULT_TABLE table=test-table [id=b165ac61b3574ee3984395cb82aef241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:51.506275 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06eefd42819411495f9ad7e328fd58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:51.506281 19710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06eefd42819411495f9ad7e328fd58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:51.506281 19576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06eefd42819411495f9ad7e328fd58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:51.508889 19793 tablet_bootstrap.cc:492] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681: Bootstrap starting.
I20260925 21:59:51.509541 19793 tablet_bootstrap.cc:654] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:51.509578 19795 tablet_bootstrap.cc:492] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8: Bootstrap starting.
I20260925 21:59:51.510135 19793 log.cc:824] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:51.510221 19795 tablet_bootstrap.cc:654] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:51.510712 19795 log.cc:824] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:51.510887 19793 tablet_bootstrap.cc:492] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681: No bootstrap required, opened a new log
I20260925 21:59:51.510959 19793 ts_tablet_manager.cc:1397] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:59:51.510994 19794 tablet_bootstrap.cc:492] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6: Bootstrap starting.
I20260925 21:59:51.511420 19795 tablet_bootstrap.cc:492] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8: No bootstrap required, opened a new log
I20260925 21:59:51.511538 19795 ts_tablet_manager.cc:1397] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:59:51.511631 19794 tablet_bootstrap.cc:654] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:51.512204 19794 log.cc:824] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:51.512385 19793 raft_consensus.cc:348] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af73652e01f74bc4b4080b66317e8aa8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33509 } } peers { permanent_uuid: "1495070baff9438faebb3677f5c7ebb6" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39585 } } peers { permanent_uuid: "47a1633350d04fffa6c77cb103f40681" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45085 } }
I20260925 21:59:51.512578 19793 raft_consensus.cc:374] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:51.512629 19793 raft_consensus.cc:729] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a1633350d04fffa6c77cb103f40681, State: Initialized, Role: FOLLOWER
I20260925 21:59:51.512753 19793 consensus_queue.cc:260] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af73652e01f74bc4b4080b66317e8aa8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33509 } } peers { permanent_uuid: "1495070baff9438faebb3677f5c7ebb6" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39585 } } peers { permanent_uuid: "47a1633350d04fffa6c77cb103f40681" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45085 } }
I20260925 21:59:51.513016 19794 tablet_bootstrap.cc:492] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6: No bootstrap required, opened a new log
I20260925 21:59:51.513026 19793 ts_tablet_manager.cc:1428] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:59:51.513083 19794 ts_tablet_manager.cc:1397] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 21:59:51.513146 19508 heartbeater.cc:503] Master 127.2.70.254:43019 was elected leader, sending a full tablet report...
I20260925 21:59:51.513646 19795 raft_consensus.cc:348] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af73652e01f74bc4b4080b66317e8aa8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33509 } } peers { permanent_uuid: "1495070baff9438faebb3677f5c7ebb6" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39585 } } peers { permanent_uuid: "47a1633350d04fffa6c77cb103f40681" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45085 } }
I20260925 21:59:51.513808 19795 raft_consensus.cc:374] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:51.513841 19795 raft_consensus.cc:729] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af73652e01f74bc4b4080b66317e8aa8, State: Initialized, Role: FOLLOWER
I20260925 21:59:51.513952 19795 consensus_queue.cc:260] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af73652e01f74bc4b4080b66317e8aa8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33509 } } peers { permanent_uuid: "1495070baff9438faebb3677f5c7ebb6" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39585 } } peers { permanent_uuid: "47a1633350d04fffa6c77cb103f40681" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45085 } }
I20260925 21:59:51.514371 19795 ts_tablet_manager.cc:1428] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 21:59:51.514415 19642 heartbeater.cc:503] Master 127.2.70.254:43019 was elected leader, sending a full tablet report...
I20260925 21:59:51.514772 19794 raft_consensus.cc:348] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af73652e01f74bc4b4080b66317e8aa8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33509 } } peers { permanent_uuid: "1495070baff9438faebb3677f5c7ebb6" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39585 } } peers { permanent_uuid: "47a1633350d04fffa6c77cb103f40681" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45085 } }
I20260925 21:59:51.514940 19794 raft_consensus.cc:374] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:51.514974 19794 raft_consensus.cc:729] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1495070baff9438faebb3677f5c7ebb6, State: Initialized, Role: FOLLOWER
I20260925 21:59:51.515082 19794 consensus_queue.cc:260] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af73652e01f74bc4b4080b66317e8aa8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33509 } } peers { permanent_uuid: "1495070baff9438faebb3677f5c7ebb6" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39585 } } peers { permanent_uuid: "47a1633350d04fffa6c77cb103f40681" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45085 } }
I20260925 21:59:51.515339 19794 ts_tablet_manager.cc:1428] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 21:59:51.515367 19776 heartbeater.cc:503] Master 127.2.70.254:43019 was elected leader, sending a full tablet report...
W20260925 21:59:51.565016 19643 tablet.cc:2369] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:59:51.627696 19801 raft_consensus.cc:482] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:59:51.627875 19801 raft_consensus.cc:504] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af73652e01f74bc4b4080b66317e8aa8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33509 } } peers { permanent_uuid: "1495070baff9438faebb3677f5c7ebb6" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39585 } } peers { permanent_uuid: "47a1633350d04fffa6c77cb103f40681" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45085 } }
I20260925 21:59:51.628244 19801 leader_election.cc:288] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af73652e01f74bc4b4080b66317e8aa8 (127.2.70.194:33509), 47a1633350d04fffa6c77cb103f40681 (127.2.70.193:45085)
I20260925 21:59:51.631193 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.628507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:38163 (local address 127.2.70.194:33509)
0925 21:59:51.628561 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:51.628562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:51.628574 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:59:51.628609 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:51.628610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:51.628624 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:51.628704 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:51.628855 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.629473 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.630419 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.630844 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.630911 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.630921 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:51.631073 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 21:59:51.631116 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 21:59:51.631198 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.628627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:59737 (local address 127.2.70.193:45085)
0925 21:59:51.628692 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:51.628694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:51.628797 (+ 103us) server_negotiation.cc:407] Connection header received
0925 21:59:51.628852 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:51.628853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:51.628866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:51.628906 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:51.629104 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.629551 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.630510 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.630881 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.630948 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.630956 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:51.631096 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 21:59:51.631133 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 21:59:51.631210 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.628425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:33509 (local address 127.2.70.195:38163)
0925 21:59:51.628476 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:51.628488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:51.628533 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:51.628711 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:51.628712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:51.628715 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:51.628783 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.628785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.629558 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.629559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.630326 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.630328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.630930 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.630931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.631006 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.631009 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:51.631053 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 21:59:51.631098 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":11}
I20260925 21:59:51.631222 19802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.628522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:45085 (local address 127.2.70.195:59737)
0925 21:59:51.628727 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:51.628738 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:51.628798 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:51.628928 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:51.628929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:51.628931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:51.629001 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.629003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.629631 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.629632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.630423 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.630425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.630966 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.630967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.631023 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.631027 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:59:51.631084 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 21:59:51.631121 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":126,"threads_started":1}
I20260925 21:59:51.631531 19462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06eefd42819411495f9ad7e328fd58d" candidate_uuid: "1495070baff9438faebb3677f5c7ebb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a1633350d04fffa6c77cb103f40681" is_pre_election: true
I20260925 21:59:51.631551 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06eefd42819411495f9ad7e328fd58d" candidate_uuid: "1495070baff9438faebb3677f5c7ebb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af73652e01f74bc4b4080b66317e8aa8" is_pre_election: true
I20260925 21:59:51.631701 19462 raft_consensus.cc:2454] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1495070baff9438faebb3677f5c7ebb6 in term 0.
I20260925 21:59:51.631701 19596 raft_consensus.cc:2454] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1495070baff9438faebb3677f5c7ebb6 in term 0.
I20260925 21:59:51.631933 19666 leader_election.cc:302] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1495070baff9438faebb3677f5c7ebb6, af73652e01f74bc4b4080b66317e8aa8; no voters:
I20260925 21:59:51.632054 19801 raft_consensus.cc:2781] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:59:51.632128 19801 raft_consensus.cc:482] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:59:51.632154 19801 raft_consensus.cc:3037] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:51.632894 19801 raft_consensus.cc:504] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af73652e01f74bc4b4080b66317e8aa8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33509 } } peers { permanent_uuid: "1495070baff9438faebb3677f5c7ebb6" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39585 } } peers { permanent_uuid: "47a1633350d04fffa6c77cb103f40681" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45085 } }
I20260925 21:59:51.633025 19801 leader_election.cc:288] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [CANDIDATE]: Term 1 election: Requested vote from peers af73652e01f74bc4b4080b66317e8aa8 (127.2.70.194:33509), 47a1633350d04fffa6c77cb103f40681 (127.2.70.193:45085)
I20260925 21:59:51.633206 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06eefd42819411495f9ad7e328fd58d" candidate_uuid: "1495070baff9438faebb3677f5c7ebb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af73652e01f74bc4b4080b66317e8aa8"
I20260925 21:59:51.633229 19462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06eefd42819411495f9ad7e328fd58d" candidate_uuid: "1495070baff9438faebb3677f5c7ebb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a1633350d04fffa6c77cb103f40681"
I20260925 21:59:51.633291 19596 raft_consensus.cc:3037] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:51.633296 19462 raft_consensus.cc:3037] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:51.634162 19596 raft_consensus.cc:2454] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1495070baff9438faebb3677f5c7ebb6 in term 1.
I20260925 21:59:51.634162 19462 raft_consensus.cc:2454] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1495070baff9438faebb3677f5c7ebb6 in term 1.
I20260925 21:59:51.634359 19666 leader_election.cc:302] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1495070baff9438faebb3677f5c7ebb6, 47a1633350d04fffa6c77cb103f40681; no voters:
I20260925 21:59:51.634439 19801 raft_consensus.cc:2781] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:51.634601 19801 raft_consensus.cc:686] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 1 LEADER]: Becoming Leader. State: Replica: 1495070baff9438faebb3677f5c7ebb6, State: Running, Role: LEADER
I20260925 21:59:51.634683 19801 consensus_queue.cc:237] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af73652e01f74bc4b4080b66317e8aa8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33509 } } peers { permanent_uuid: "1495070baff9438faebb3677f5c7ebb6" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39585 } } peers { permanent_uuid: "47a1633350d04fffa6c77cb103f40681" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45085 } }
I20260925 21:59:51.635522 19316 catalog_manager.cc:5495] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1495070baff9438faebb3677f5c7ebb6 (127.2.70.195). New cstate: current_term: 1 leader_uuid: "1495070baff9438faebb3677f5c7ebb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af73652e01f74bc4b4080b66317e8aa8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1495070baff9438faebb3677f5c7ebb6" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 39585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47a1633350d04fffa6c77cb103f40681" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45085 } health_report { overall_health: UNKNOWN } } }
I20260925 21:59:51.652292 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.649912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.2.70.195:39585)
0925 21:59:51.650018 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:51.650020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:51.650028 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:59:51.650067 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:51.650067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:51.650085 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:51.650143 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:59:51.650268 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.650754 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.651529 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.651717 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.651769 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.651958 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:59:51.652065 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:59:51.652137 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:51.652169 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 21:59:51.652191 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 21:59:51.654100 19462 raft_consensus.cc:1261] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681 [term 1 FOLLOWER]: Refusing update from remote peer 1495070baff9438faebb3677f5c7ebb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:51.654100 19596 raft_consensus.cc:1261] T e06eefd42819411495f9ad7e328fd58d P af73652e01f74bc4b4080b66317e8aa8 [term 1 FOLLOWER]: Refusing update from remote peer 1495070baff9438faebb3677f5c7ebb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:51.654424 19803 consensus_queue.cc:1035] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af73652e01f74bc4b4080b66317e8aa8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 33509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:51.654546 19801 consensus_queue.cc:1035] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a1633350d04fffa6c77cb103f40681" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 45085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:51.657346 19808 mvcc.cc:205] Tried to move back new op lower bound from 7333370231412580352 to 7333370231336300544. Current Snapshot: MvccSnapshot[applied={T|T < 7333370231412580352}]
I20260925 21:59:51.657639 19807 mvcc.cc:205] Tried to move back new op lower bound from 7333370231412580352 to 7333370231336300544. Current Snapshot: MvccSnapshot[applied={T|T < 7333370231412580352}]
I20260925 21:59:51.658926 19317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:51.659022 19317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:51.659767 19317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43158a616ca640f38469676004a890b2: Sending DeleteTablet for 3 replicas of tablet e06eefd42819411495f9ad7e328fd58d
I20260925 21:59:51.660111 19710 tablet_service.cc:1514] Processing DeleteTablet for tablet e06eefd42819411495f9ad7e328fd58d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:51 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:45442
I20260925 21:59:51.660144 19576 tablet_service.cc:1514] Processing DeleteTablet for tablet e06eefd42819411495f9ad7e328fd58d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:51 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:51402
I20260925 21:59:51.660261 19442 tablet_service.cc:1514] Processing DeleteTablet for tablet e06eefd42819411495f9ad7e328fd58d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:51 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:60712
I20260925 21:59:51.660471 19817 tablet_replica.cc:330] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681: stopping tablet replica
I20260925 21:59:51.660526 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 19379
I20260925 21:59:51.660561 19817 raft_consensus.cc:2229] T e06eefd42819411495f9ad7e328fd58d P 47a1633350d04fffa6c77cb103f40681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:59:51.662022 19815 tablet_replica.cc:330] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6: stopping tablet replica
I20260925 21:59:51.662122 19815 raft_consensus.cc:2229] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:59:51.662209 19815 raft_consensus.cc:2258] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:51.662624 19815 ts_tablet_manager.cc:1905] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:51.664737 19815 ts_tablet_manager.cc:1918] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:51.664808 19815 log.cc:1196] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/e06eefd42819411495f9ad7e328fd58d
I20260925 21:59:51.665041 19815 ts_tablet_manager.cc:1939] T e06eefd42819411495f9ad7e328fd58d P 1495070baff9438faebb3677f5c7ebb6: Deleting consensus metadata
I20260925 21:59:51.665410 19304 catalog_manager.cc:4841] TS 1495070baff9438faebb3677f5c7ebb6 (127.2.70.195:39585): tablet e06eefd42819411495f9ad7e328fd58d (table test-table [id=b165ac61b3574ee3984395cb82aef241]) successfully deleted
W20260925 21:59:51.666221 19304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:45085 (error 108)
I20260925 21:59:51.666741 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.666616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:45085 (local address 127.0.0.1:60720)
0925 21:59:51.666670 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:51.666707 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 21:59:51.666749 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 19513
W20260925 21:59:51.669986 19304 catalog_manager.cc:4584] TS 47a1633350d04fffa6c77cb103f40681 (127.2.70.193:45085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e06eefd42819411495f9ad7e328fd58d: Network error: Client connection negotiation failed: client connection to 127.2.70.193:45085: connect: Connection refused (error 111)
I20260925 21:59:51.674098 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 19647
I20260925 21:59:51.674136 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.674024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:33509 (local address 127.0.0.1:51416)
0925 21:59:51.674069 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:51.674101 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.194:33509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 21:59:51.677968 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 19285
2026-09-25T21:59:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1951 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T21:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:59:51Z Disabled control of system clock
I20260925 21:59:51.703604 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:34487
--webserver_interface=127.2.70.254
--webserver_port=0
--builtin_ntp_servers=127.2.70.212:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 21:59:51.784904 19822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:51.785151 19822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:51.785171 19822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:51.786329 19822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:59:51.786365 19822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:51.786376 19822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:51.786423 19822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:59:51.786434 19822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:59:51.787868 19822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:34487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:51.788132 19822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:51.788415 19822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:51.793213 19830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:51.793244 19827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:51.793207 19828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:51.793537 19822 server_base.cc:1034] running on GCE node
I20260925 21:59:51.793792 19822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:51.794085 19822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:51.795250 19822 hybrid_clock.cc:648] HybridClock initialized: now 1790373591795232 us; error 31 us; skew 500 ppm
I20260925 21:59:51.796504 19822 webserver.cc:460] Webserver started at http://127.2.70.254:39049/ using document root <none> and password file <none>
I20260925 21:59:51.796737 19822 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:51.796787 19822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:51.796914 19822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:51.798022 19822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "dcdba77dd06844ab9a454a01c5435294"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "40434432768e57c56af3de612eb280db"
server_key_iv: "920ec07557587d928da420c46c3548d1"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.798348 19822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "dcdba77dd06844ab9a454a01c5435294"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "40434432768e57c56af3de612eb280db"
server_key_iv: "920ec07557587d928da420c46c3548d1"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.799558 19822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 21:59:51.800323 19836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.800493 19822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.800601 19822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "dcdba77dd06844ab9a454a01c5435294"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "40434432768e57c56af3de612eb280db"
server_key_iv: "920ec07557587d928da420c46c3548d1"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.800661 19822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:51.821195 19822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:51.821519 19822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:51.821655 19822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:51.825505 19822 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:34487
I20260925 21:59:51.825559 19888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:34487 every 8 connection(s)
I20260925 21:59:51.825886 19822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:59:51.826858 19889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:51.828228 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 19822
I20260925 21:59:51.828318 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:59:51.828507 2331 external_mini_cluster.cc:1428] Setting key 6a696e185ca47def40d9f44b0498aaf1
I20260925 21:59:51.829640 19889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294: Bootstrap starting.
I20260925 21:59:51.830340 19889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:51.830686 19889 log.cc:824] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:51.831413 19889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294: No bootstrap required, opened a new log
I20260925 21:59:51.833506 19889 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdba77dd06844ab9a454a01c5435294" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34487 } }
I20260925 21:59:51.833679 19889 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:51.833715 19889 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdba77dd06844ab9a454a01c5435294, State: Initialized, Role: FOLLOWER
I20260925 21:59:51.833859 19889 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdba77dd06844ab9a454a01c5435294" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34487 } }
I20260925 21:59:51.833940 19889 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:59:51.833987 19889 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:59:51.834033 19889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:51.834429 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.829088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.2.70.254:34487)
0925 21:59:51.829308 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:51.829318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:51.829350 (+ 32us) server_negotiation.cc:407] Connection header received
0925 21:59:51.829874 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:51.829883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:51.829966 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:51.830140 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:51.830317 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.831608 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.832420 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.833040 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.833172 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.833755 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:51.833760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:51.833760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:51.833956 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:51.834000 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:51.834014 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:51.834077 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:51.834145 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 21:59:51.834224 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260925 21:59:51.834678 19889 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdba77dd06844ab9a454a01c5435294" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34487 } }
I20260925 21:59:51.834774 19889 leader_election.cc:302] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcdba77dd06844ab9a454a01c5435294; no voters:
I20260925 21:59:51.834928 19889 leader_election.cc:288] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:59:51.834959 19894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:51.835057 19894 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [term 1 LEADER]: Becoming Leader. State: Replica: dcdba77dd06844ab9a454a01c5435294, State: Running, Role: LEADER
I20260925 21:59:51.835188 19889 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:59:51.835196 19894 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdba77dd06844ab9a454a01c5435294" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34487 } }
I20260925 21:59:51.835609 19895 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcdba77dd06844ab9a454a01c5435294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdba77dd06844ab9a454a01c5435294" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34487 } } }
I20260925 21:59:51.835620 19896 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcdba77dd06844ab9a454a01c5435294. Latest consensus state: current_term: 1 leader_uuid: "dcdba77dd06844ab9a454a01c5435294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdba77dd06844ab9a454a01c5435294" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34487 } } }
I20260925 21:59:51.835781 19895 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:51.835801 19896 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294 [sys.catalog]: This master's current role is: LEADER
W20260925 21:59:51.836764 19910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:59:51.836805 19910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:59:51.836895 19904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:59:51.837077 19904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:59:51.838466 19904 catalog_manager.cc:1347] Generated new cluster ID: a573d5d5d304404b8e94bd2b23e53d75
I20260925 21:59:51.838514 19904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:59:51.843504 19904 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:59:51.844259 19904 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:59:51.861179 19904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294: Generated new TSK 0
I20260925 21:59:51.861423 19904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:59:51.864879 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34487
--builtin_ntp_servers=127.2.70.212:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 21:59:51.945531 19913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:51.945748 19913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:51.945776 19913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:51.947000 19913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:51.947043 19913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:51.947084 19913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:59:51.948540 19913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:51.948818 19913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:51.949115 19913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:51.954002 19918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:51.954051 19921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:51.954027 19919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:51.954366 19913 server_base.cc:1034] running on GCE node
I20260925 21:59:51.954674 19913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:51.954910 19913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:51.956048 19913 hybrid_clock.cc:648] HybridClock initialized: now 1790373591956028 us; error 38 us; skew 500 ppm
I20260925 21:59:51.957289 19913 webserver.cc:460] Webserver started at http://127.2.70.193:33349/ using document root <none> and password file <none>
I20260925 21:59:51.957516 19913 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:51.957588 19913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:51.957720 19913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:51.958745 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "d6d0c3ec959a4614bbce375e624caaa6"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "8e0ae9e952ca634b2a9e660868b6bb80"
server_key_iv: "f9ed03679a2b8a347544e6e036209738"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.959102 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "d6d0c3ec959a4614bbce375e624caaa6"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "8e0ae9e952ca634b2a9e660868b6bb80"
server_key_iv: "f9ed03679a2b8a347544e6e036209738"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.960418 19913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:59:51.961248 19927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.961413 19913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:51.961495 19913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "d6d0c3ec959a4614bbce375e624caaa6"
format_stamp: "Formatted at 2026-09-25 21:59:51 on dist-test-slave-2g0f"
server_key: "8e0ae9e952ca634b2a9e660868b6bb80"
server_key_iv: "f9ed03679a2b8a347544e6e036209738"
server_key_version: "encryptionkey@0"
I20260925 21:59:51.961560 19913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:51.978826 19913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:51.979158 19913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:51.979307 19913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:51.979525 19913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:51.979894 19913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:51.979928 19913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.979956 19913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:51.979979 19913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:51.985245 19913 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:43435
I20260925 21:59:51.985314 20040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:43435 every 8 connection(s)
I20260925 21:59:51.985621 19913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:59:51.989408 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 19913
I20260925 21:59:51.989513 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:59:51.989714 2331 external_mini_cluster.cc:1428] Setting key a420c3c378e0496100b44c22429c91aa
I20260925 21:59:51.990658 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34487
--builtin_ntp_servers=127.2.70.212:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 21:59:51.990914 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.986315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:45277 (local address 127.2.70.254:34487)
0925 21:59:51.986370 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:51.986372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:51.986614 (+ 242us) server_negotiation.cc:407] Connection header received
0925 21:59:51.987102 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:51.987103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:51.987116 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:51.987176 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:51.987796 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.988342 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.989405 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.989717 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.989798 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.990639 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:51.990640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:51.990640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:51.990645 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:51.990660 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:51.990663 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:51.990711 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:51.990808 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 21:59:51.990834 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 21:59:51.991269 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:51.986292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34487 (local address 127.2.70.193:45277)
0925 21:59:51.986497 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:51.986537 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:51.987012 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:51.987230 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:51.987234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:51.987242 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:51.987735 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.987739 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.988462 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.988463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.989306 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:51.989309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:51.989814 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:51.989815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:51.989951 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:51.990491 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:59:51.990491 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:51.990520 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:59:51.990546 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:51.990717 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:51.990722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:51.990725 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:51.991005 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 21:59:51.991112 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260925 21:59:51.991698 20041 heartbeater.cc:347] Connected to a master server at 127.2.70.254:34487
I20260925 21:59:51.991794 20041 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:51.992022 20041 heartbeater.cc:511] Master 127.2.70.254:34487 requested a full tablet report, sending...
I20260925 21:59:51.992519 19853 ts_manager.cc:194] Registered new tserver with Master: d6d0c3ec959a4614bbce375e624caaa6 (127.2.70.193:43435)
I20260925 21:59:51.993127 19853 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.193:45277
W20260925 21:59:52.073599 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:52.073827 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:52.073858 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:52.075109 20044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:52.075153 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:52.075193 20044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:59:52.076660 20044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:52.076897 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:52.077215 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:52.082108 20050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:52.082125 20049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:52.082125 20052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:52.082470 20044 server_base.cc:1034] running on GCE node
I20260925 21:59:52.082659 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:52.082854 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:52.083992 20044 hybrid_clock.cc:648] HybridClock initialized: now 1790373592083974 us; error 28 us; skew 500 ppm
I20260925 21:59:52.085212 20044 webserver.cc:460] Webserver started at http://127.2.70.194:40395/ using document root <none> and password file <none>
I20260925 21:59:52.085464 20044 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:52.085515 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:52.085636 20044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:52.086678 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "dbfa0447011f411d90458b3301970ea3"
format_stamp: "Formatted at 2026-09-25 21:59:52 on dist-test-slave-2g0f"
server_key: "f1c15609212f0a63961a281e00e4e4ba"
server_key_iv: "1b40ba9426270ad889b15958395460ff"
server_key_version: "encryptionkey@0"
I20260925 21:59:52.086995 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "dbfa0447011f411d90458b3301970ea3"
format_stamp: "Formatted at 2026-09-25 21:59:52 on dist-test-slave-2g0f"
server_key: "f1c15609212f0a63961a281e00e4e4ba"
server_key_iv: "1b40ba9426270ad889b15958395460ff"
server_key_version: "encryptionkey@0"
I20260925 21:59:52.088232 20044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:59:52.088876 20058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:52.089021 20044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:52.089099 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "dbfa0447011f411d90458b3301970ea3"
format_stamp: "Formatted at 2026-09-25 21:59:52 on dist-test-slave-2g0f"
server_key: "f1c15609212f0a63961a281e00e4e4ba"
server_key_iv: "1b40ba9426270ad889b15958395460ff"
server_key_version: "encryptionkey@0"
I20260925 21:59:52.089161 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:52.111446 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:52.111774 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:52.111927 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:52.112149 20044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:52.112466 20044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:52.112500 20044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:52.112530 20044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:52.112546 20044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:52.117702 20044 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:42499
I20260925 21:59:52.117763 20171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:42499 every 8 connection(s)
I20260925 21:59:52.118063 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:59:52.122929 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.118732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:54649 (local address 127.2.70.254:34487)
0925 21:59:52.118816 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:52.118818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:52.119009 (+ 191us) server_negotiation.cc:407] Connection header received
0925 21:59:52.119465 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:52.119466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:52.119476 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:52.119527 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:52.120250 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.120739 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.121728 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.121918 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.121984 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.122715 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:52.122717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:52.122717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:52.122721 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:52.122735 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:52.122738 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:52.122786 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:52.122847 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 21:59:52.122869 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 21:59:52.123123 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.118718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34487 (local address 127.2.70.194:54649)
0925 21:59:52.118894 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:52.118928 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:52.119389 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:52.119584 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:52.119587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:52.119596 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:52.120162 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.120167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.120810 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.120811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.121655 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.121658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.121990 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.121991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.122113 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.122583 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:59:52.122583 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:52.122611 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:59:52.122632 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:52.122790 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:52.122793 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:52.122795 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:59:52.122839 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 21:59:52.122914 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260925 21:59:52.123569 20172 heartbeater.cc:347] Connected to a master server at 127.2.70.254:34487
I20260925 21:59:52.123653 20172 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:52.123799 20172 heartbeater.cc:511] Master 127.2.70.254:34487 requested a full tablet report, sending...
I20260925 21:59:52.124171 19853 ts_manager.cc:194] Registered new tserver with Master: dbfa0447011f411d90458b3301970ea3 (127.2.70.194:42499)
I20260925 21:59:52.124521 19853 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.194:54649
I20260925 21:59:52.126464 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 20044
I20260925 21:59:52.126546 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:59:52.126708 2331 external_mini_cluster.cc:1428] Setting key dbeb7c230b052049bc3002342acece90
I20260925 21:59:52.127506 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34487
--builtin_ntp_servers=127.2.70.212:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 21:59:52.206804 20175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:52.207020 20175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:52.207041 20175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:52.208215 20175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:52.208256 20175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:52.208281 20175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:59:52.209630 20175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:52.209851 20175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:52.210134 20175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:52.215152 20181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:52.215119 20183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:52.215143 20180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:52.215166 20175 server_base.cc:1034] running on GCE node
I20260925 21:59:52.215548 20175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:52.215761 20175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:52.216904 20175 hybrid_clock.cc:648] HybridClock initialized: now 1790373592216892 us; error 30 us; skew 500 ppm
I20260925 21:59:52.218019 20175 webserver.cc:460] Webserver started at http://127.2.70.195:38501/ using document root <none> and password file <none>
I20260925 21:59:52.218238 20175 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:52.218286 20175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:52.218426 20175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:52.219467 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "aace8997dc564485a5aee5b0354ac84a"
format_stamp: "Formatted at 2026-09-25 21:59:52 on dist-test-slave-2g0f"
server_key: "7f6fcbc49f0f952c78b82913181c08e9"
server_key_iv: "37ceb7e2fcead1e2e9850655c8678d7c"
server_key_version: "encryptionkey@0"
I20260925 21:59:52.219784 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "aace8997dc564485a5aee5b0354ac84a"
format_stamp: "Formatted at 2026-09-25 21:59:52 on dist-test-slave-2g0f"
server_key: "7f6fcbc49f0f952c78b82913181c08e9"
server_key_iv: "37ceb7e2fcead1e2e9850655c8678d7c"
server_key_version: "encryptionkey@0"
I20260925 21:59:52.221012 20175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:52.221733 20189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:52.221901 20175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:52.221982 20175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "aace8997dc564485a5aee5b0354ac84a"
format_stamp: "Formatted at 2026-09-25 21:59:52 on dist-test-slave-2g0f"
server_key: "7f6fcbc49f0f952c78b82913181c08e9"
server_key_iv: "37ceb7e2fcead1e2e9850655c8678d7c"
server_key_version: "encryptionkey@0"
I20260925 21:59:52.222056 20175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:52.243748 20175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:52.244103 20175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:52.244242 20175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:52.244479 20175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:52.244797 20175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:52.244830 20175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:52.244860 20175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:52.244876 20175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:52.250190 20175 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:33651
I20260925 21:59:52.250241 20302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:33651 every 8 connection(s)
I20260925 21:59:52.250559 20175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:59:52.252209 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 20175
I20260925 21:59:52.252310 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:59:52.252496 2331 external_mini_cluster.cc:1428] Setting key 5545e1eeb525bf0652920339323622c3
I20260925 21:59:52.255761 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.251231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:42163 (local address 127.2.70.254:34487)
0925 21:59:52.251296 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:52.251297 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:52.251528 (+ 231us) server_negotiation.cc:407] Connection header received
0925 21:59:52.252031 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:52.252032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:52.252041 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:52.252100 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:52.252933 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.253561 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.254522 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.254712 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.254773 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.255531 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:52.255532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:52.255533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:52.255537 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:52.255552 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:52.255555 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:52.255597 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:52.255664 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 21:59:52.255687 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 21:59:52.255965 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.251220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34487 (local address 127.2.70.195:42163)
0925 21:59:52.251400 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:52.251442 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:52.251939 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:52.252194 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:52.252199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:52.252210 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:52.252839 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.252846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.253633 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.253633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.254426 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.254429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.254779 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.254780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.254883 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.255388 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:59:52.255388 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:52.255419 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:59:52.255442 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:52.255599 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:52.255601 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:52.255604 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:52.255649 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 21:59:52.255731 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260925 21:59:52.256346 20303 heartbeater.cc:347] Connected to a master server at 127.2.70.254:34487
I20260925 21:59:52.256434 20303 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:52.256584 20303 heartbeater.cc:511] Master 127.2.70.254:34487 requested a full tablet report, sending...
I20260925 21:59:52.256904 19853 ts_manager.cc:194] Registered new tserver with Master: aace8997dc564485a5aee5b0354ac84a (127.2.70.195:33651)
I20260925 21:59:52.257237 19853 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.195:42163
I20260925 21:59:52.264379 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:59:52.268537 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.265785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.2.70.254:34487)
0925 21:59:52.265840 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:52.265842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:52.265943 (+ 101us) server_negotiation.cc:407] Connection header received
0925 21:59:52.266009 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:52.266010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:52.266020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:52.266066 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:52.266209 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.266696 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.267419 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.267629 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.267682 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.268308 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:52.268310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:52.268310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:52.268315 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:52.268330 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:52.268334 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:52.268374 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:52.268431 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 21:59:52.268454 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 21:59:52.270133 19853 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:59:52.271603 19853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:59:52.278085 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.273721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:42499 (local address 127.0.0.1:39690)
0925 21:59:52.273903 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:52.273945 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:52.274010 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:52.274793 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:52.274794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:52.274797 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:52.274839 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.274840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.275692 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.275693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.277064 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.277067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.277769 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.277770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.277848 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.277862 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 21:59:52.277933 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 21:59:52.277993 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":65,"thread_start_us":72,"threads_started":1}
I20260925 21:59:52.278173 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.273894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.2.70.193:43435)
0925 21:59:52.274146 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:52.274152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:52.274164 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:59:52.274207 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:52.274209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:52.274221 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:52.274290 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:52.274475 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.275663 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.277157 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.277845 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.277895 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.277979 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:52.278052 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 21:59:52.278089 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":137,"threads_started":1}
I20260925 21:59:52.278258 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.273993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.2.70.195:33651)
0925 21:59:52.274371 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:52.274376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:52.274408 (+ 32us) server_negotiation.cc:407] Connection header received
0925 21:59:52.274450 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:52.274452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:52.274466 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:52.274532 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:52.274962 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.276002 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.277020 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.277751 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.277819 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.277907 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:52.278138 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 21:59:52.278181 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":86,"threads_started":1}
I20260925 21:59:52.278266 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.273984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39690 (local address 127.2.70.194:42499)
0925 21:59:52.274658 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:52.274663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:52.274672 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:59:52.274707 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:52.274709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:52.274724 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:52.274793 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:52.274917 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.275608 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.277180 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.277692 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.277757 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.277856 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:52.278137 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 21:59:52.278179 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":88,"threads_started":1}
I20260925 21:59:52.278532 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.273721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:43435 (local address 127.0.0.1:58380)
0925 21:59:52.274111 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:52.274119 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:52.274154 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:52.274316 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:52.274319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:52.274321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:52.274393 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.274395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.275737 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.275738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.277074 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.277075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.277917 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.277918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.277990 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.277993 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:52.278038 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 21:59:52.278075 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":32,"thread_start_us":83,"threads_started":1}
I20260925 21:59:52.278474 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.273725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:33651 (local address 127.0.0.1:38532)
0925 21:59:52.274216 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:52.274222 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:52.274247 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:52.274850 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:52.274850 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:52.274852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:52.274898 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.274900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.276069 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.276070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.276946 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.276949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.277945 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.277947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.278008 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.278012 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:59:52.278060 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 21:59:52.278099 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":109,"threads_started":1}
I20260925 21:59:52.279275 20106 tablet_service.cc:1467] Processing CreateTablet for tablet f53551564691418b8c8e338763903844 (DEFAULT_TABLE table=test-table [id=44cbe33f885a43b5a6b98b299b05d901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:52.279275 19975 tablet_service.cc:1467] Processing CreateTablet for tablet f53551564691418b8c8e338763903844 (DEFAULT_TABLE table=test-table [id=44cbe33f885a43b5a6b98b299b05d901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:52.279646 19975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53551564691418b8c8e338763903844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:52.279646 20106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53551564691418b8c8e338763903844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:52.280164 20237 tablet_service.cc:1467] Processing CreateTablet for tablet f53551564691418b8c8e338763903844 (DEFAULT_TABLE table=test-table [id=44cbe33f885a43b5a6b98b299b05d901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:52.280447 20237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53551564691418b8c8e338763903844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:52.282896 20317 tablet_bootstrap.cc:492] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a: Bootstrap starting.
I20260925 21:59:52.283164 20319 tablet_bootstrap.cc:492] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3: Bootstrap starting.
I20260925 21:59:52.283528 20317 tablet_bootstrap.cc:654] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:52.283674 20319 tablet_bootstrap.cc:654] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:52.283803 20317 log.cc:824] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:52.283978 20318 tablet_bootstrap.cc:492] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6: Bootstrap starting.
I20260925 21:59:52.283991 20319 log.cc:824] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:52.284528 20318 tablet_bootstrap.cc:654] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:52.284616 20317 tablet_bootstrap.cc:492] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a: No bootstrap required, opened a new log
I20260925 21:59:52.284689 20319 tablet_bootstrap.cc:492] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3: No bootstrap required, opened a new log
I20260925 21:59:52.284689 20317 ts_tablet_manager.cc:1397] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:59:52.284744 20319 ts_tablet_manager.cc:1397] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 21:59:52.284829 20318 log.cc:824] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:52.285558 20318 tablet_bootstrap.cc:492] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6: No bootstrap required, opened a new log
I20260925 21:59:52.285617 20318 ts_tablet_manager.cc:1397] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:59:52.287119 20319 raft_consensus.cc:348] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d0c3ec959a4614bbce375e624caaa6" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43435 } } peers { permanent_uuid: "aace8997dc564485a5aee5b0354ac84a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 33651 } } peers { permanent_uuid: "dbfa0447011f411d90458b3301970ea3" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42499 } }
I20260925 21:59:52.287119 20317 raft_consensus.cc:348] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d0c3ec959a4614bbce375e624caaa6" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43435 } } peers { permanent_uuid: "aace8997dc564485a5aee5b0354ac84a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 33651 } } peers { permanent_uuid: "dbfa0447011f411d90458b3301970ea3" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42499 } }
I20260925 21:59:52.287292 20317 raft_consensus.cc:374] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:52.287293 20319 raft_consensus.cc:374] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:52.287329 20317 raft_consensus.cc:729] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aace8997dc564485a5aee5b0354ac84a, State: Initialized, Role: FOLLOWER
I20260925 21:59:52.287329 20319 raft_consensus.cc:729] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbfa0447011f411d90458b3301970ea3, State: Initialized, Role: FOLLOWER
I20260925 21:59:52.287285 20318 raft_consensus.cc:348] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d0c3ec959a4614bbce375e624caaa6" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43435 } } peers { permanent_uuid: "aace8997dc564485a5aee5b0354ac84a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 33651 } } peers { permanent_uuid: "dbfa0447011f411d90458b3301970ea3" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42499 } }
I20260925 21:59:52.287433 20318 raft_consensus.cc:374] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:52.287470 20318 raft_consensus.cc:729] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d0c3ec959a4614bbce375e624caaa6, State: Initialized, Role: FOLLOWER
I20260925 21:59:52.287475 20319 consensus_queue.cc:260] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d0c3ec959a4614bbce375e624caaa6" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43435 } } peers { permanent_uuid: "aace8997dc564485a5aee5b0354ac84a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 33651 } } peers { permanent_uuid: "dbfa0447011f411d90458b3301970ea3" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42499 } }
I20260925 21:59:52.287475 20317 consensus_queue.cc:260] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d0c3ec959a4614bbce375e624caaa6" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43435 } } peers { permanent_uuid: "aace8997dc564485a5aee5b0354ac84a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 33651 } } peers { permanent_uuid: "dbfa0447011f411d90458b3301970ea3" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42499 } }
I20260925 21:59:52.287575 20318 consensus_queue.cc:260] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d0c3ec959a4614bbce375e624caaa6" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43435 } } peers { permanent_uuid: "aace8997dc564485a5aee5b0354ac84a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 33651 } } peers { permanent_uuid: "dbfa0447011f411d90458b3301970ea3" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42499 } }
I20260925 21:59:52.287729 20319 ts_tablet_manager.cc:1428] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 21:59:52.287729 20317 ts_tablet_manager.cc:1428] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:59:52.287814 20318 ts_tablet_manager.cc:1428] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:59:52.287866 20303 heartbeater.cc:503] Master 127.2.70.254:34487 was elected leader, sending a full tablet report...
I20260925 21:59:52.287876 20172 heartbeater.cc:503] Master 127.2.70.254:34487 was elected leader, sending a full tablet report...
I20260925 21:59:52.288115 20041 heartbeater.cc:503] Master 127.2.70.254:34487 was elected leader, sending a full tablet report...
W20260925 21:59:52.368582 20173 tablet.cc:2369] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:59:52.452310 20324 raft_consensus.cc:482] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:59:52.452457 20324 raft_consensus.cc:504] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d0c3ec959a4614bbce375e624caaa6" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43435 } } peers { permanent_uuid: "aace8997dc564485a5aee5b0354ac84a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 33651 } } peers { permanent_uuid: "dbfa0447011f411d90458b3301970ea3" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42499 } }
I20260925 21:59:52.452811 20324 leader_election.cc:288] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6d0c3ec959a4614bbce375e624caaa6 (127.2.70.193:43435), dbfa0447011f411d90458b3301970ea3 (127.2.70.194:42499)
I20260925 21:59:52.456555 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.453066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:35791 (local address 127.2.70.193:43435)
0925 21:59:52.453112 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:52.453114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:52.453125 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:59:52.453157 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:52.453158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:52.453170 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:52.453230 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:52.453402 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.454153 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.455617 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.456208 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.456280 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.456285 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:52.456461 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 21:59:52.456495 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 21:59:52.456555 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.453144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:35719 (local address 127.2.70.194:42499)
0925 21:59:52.453187 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:52.453189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:52.453295 (+ 106us) server_negotiation.cc:407] Connection header received
0925 21:59:52.453323 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:52.453324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:52.453332 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:52.453370 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:52.453497 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.454194 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.455617 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.456208 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.456276 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.456285 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:52.456438 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 21:59:52.456482 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 21:59:52.456560 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.452983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:43435 (local address 127.2.70.195:35791)
0925 21:59:52.453027 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:52.453039 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:52.453088 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:52.453257 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:52.453258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:52.453259 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:52.453330 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.453331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.454220 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.454220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.455521 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.455524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.456287 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.456288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.456364 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.456371 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 21:59:52.456422 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 21:59:52.456468 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":55}
I20260925 21:59:52.456565 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.453051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:42499 (local address 127.2.70.195:35719)
0925 21:59:52.453205 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:52.453213 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:52.453260 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:52.453386 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:52.453386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:52.453387 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:52.453439 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.453440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.454248 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.454248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.455521 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:52.455524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.456289 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.456290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:52.456364 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.456371 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 21:59:52.456422 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 21:59:52.456467 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":69,"thread_start_us":103,"threads_started":1}
I20260925 21:59:52.456918 19995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53551564691418b8c8e338763903844" candidate_uuid: "aace8997dc564485a5aee5b0354ac84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d0c3ec959a4614bbce375e624caaa6" is_pre_election: true
I20260925 21:59:52.457000 20126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53551564691418b8c8e338763903844" candidate_uuid: "aace8997dc564485a5aee5b0354ac84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbfa0447011f411d90458b3301970ea3" is_pre_election: true
I20260925 21:59:52.457091 19995 raft_consensus.cc:2454] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aace8997dc564485a5aee5b0354ac84a in term 0.
I20260925 21:59:52.457120 20126 raft_consensus.cc:2454] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aace8997dc564485a5aee5b0354ac84a in term 0.
I20260925 21:59:52.457315 20193 leader_election.cc:302] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aace8997dc564485a5aee5b0354ac84a, dbfa0447011f411d90458b3301970ea3; no voters:
I20260925 21:59:52.457480 20324 raft_consensus.cc:2781] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:59:52.457516 20324 raft_consensus.cc:482] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:59:52.457530 20324 raft_consensus.cc:3037] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:52.458184 20324 raft_consensus.cc:504] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d0c3ec959a4614bbce375e624caaa6" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43435 } } peers { permanent_uuid: "aace8997dc564485a5aee5b0354ac84a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 33651 } } peers { permanent_uuid: "dbfa0447011f411d90458b3301970ea3" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42499 } }
I20260925 21:59:52.458298 20324 leader_election.cc:288] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [CANDIDATE]: Term 1 election: Requested vote from peers d6d0c3ec959a4614bbce375e624caaa6 (127.2.70.193:43435), dbfa0447011f411d90458b3301970ea3 (127.2.70.194:42499)
I20260925 21:59:52.458451 19995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53551564691418b8c8e338763903844" candidate_uuid: "aace8997dc564485a5aee5b0354ac84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d0c3ec959a4614bbce375e624caaa6"
I20260925 21:59:52.458484 20126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53551564691418b8c8e338763903844" candidate_uuid: "aace8997dc564485a5aee5b0354ac84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbfa0447011f411d90458b3301970ea3"
I20260925 21:59:52.458523 19995 raft_consensus.cc:3037] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:52.458570 20126 raft_consensus.cc:3037] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:52.459182 19995 raft_consensus.cc:2454] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aace8997dc564485a5aee5b0354ac84a in term 1.
I20260925 21:59:52.459359 20190 leader_election.cc:302] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aace8997dc564485a5aee5b0354ac84a, d6d0c3ec959a4614bbce375e624caaa6; no voters:
I20260925 21:59:52.459440 20126 raft_consensus.cc:2454] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aace8997dc564485a5aee5b0354ac84a in term 1.
I20260925 21:59:52.459467 20324 raft_consensus.cc:2781] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:52.459619 20324 raft_consensus.cc:686] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 1 LEADER]: Becoming Leader. State: Replica: aace8997dc564485a5aee5b0354ac84a, State: Running, Role: LEADER
I20260925 21:59:52.459714 20324 consensus_queue.cc:237] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d0c3ec959a4614bbce375e624caaa6" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43435 } } peers { permanent_uuid: "aace8997dc564485a5aee5b0354ac84a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 33651 } } peers { permanent_uuid: "dbfa0447011f411d90458b3301970ea3" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42499 } }
I20260925 21:59:52.460527 19851 catalog_manager.cc:5495] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a reported cstate change: term changed from 0 to 1, leader changed from <none> to aace8997dc564485a5aee5b0354ac84a (127.2.70.195). New cstate: current_term: 1 leader_uuid: "aace8997dc564485a5aee5b0354ac84a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d0c3ec959a4614bbce375e624caaa6" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aace8997dc564485a5aee5b0354ac84a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 33651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbfa0447011f411d90458b3301970ea3" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42499 } health_report { overall_health: UNKNOWN } } }
W20260925 21:59:52.486258 20042 tablet.cc:2369] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 21:59:52.501116 20304 tablet.cc:2369] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:59:52.503358 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.501149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.2.70.195:33651)
0925 21:59:52.501200 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:52.501203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:52.501214 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:59:52.501287 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:52.501288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:52.501308 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:52.501365 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:59:52.501491 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.501961 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.502731 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:52.502933 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:52.502978 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:52.503087 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:59:52.503179 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:59:52.503230 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:52.503257 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 21:59:52.503270 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 21:59:52.505051 19995 raft_consensus.cc:1261] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6 [term 1 FOLLOWER]: Refusing update from remote peer aace8997dc564485a5aee5b0354ac84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:52.505059 20126 raft_consensus.cc:1261] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3 [term 1 FOLLOWER]: Refusing update from remote peer aace8997dc564485a5aee5b0354ac84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:52.505369 20324 consensus_queue.cc:1035] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6d0c3ec959a4614bbce375e624caaa6" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:52.505501 20327 consensus_queue.cc:1035] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbfa0447011f411d90458b3301970ea3" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 42499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:52.507886 20331 mvcc.cc:205] Tried to move back new op lower bound from 7333370234898276352 to 7333370234715721728. Current Snapshot: MvccSnapshot[applied={T|T < 7333370234898276352}]
I20260925 21:59:52.508142 20332 mvcc.cc:205] Tried to move back new op lower bound from 7333370234898276352 to 7333370234715721728. Current Snapshot: MvccSnapshot[applied={T|T < 7333370234898276352}]
I20260925 21:59:52.508589 20330 mvcc.cc:205] Tried to move back new op lower bound from 7333370234898276352 to 7333370234715721728. Current Snapshot: MvccSnapshot[applied={T|T < 7333370234898276352 or (T in {7333370234898276352})}]
I20260925 21:59:52.509967 19851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:52.510047 19851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:52.510783 19851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcdba77dd06844ab9a454a01c5435294: Sending DeleteTablet for 3 replicas of tablet f53551564691418b8c8e338763903844
I20260925 21:59:52.511099 20237 tablet_service.cc:1514] Processing DeleteTablet for tablet f53551564691418b8c8e338763903844 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:52 UTC) from {username='slave'} at 127.0.0.1:38532
I20260925 21:59:52.511173 20106 tablet_service.cc:1514] Processing DeleteTablet for tablet f53551564691418b8c8e338763903844 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:52 UTC) from {username='slave'} at 127.0.0.1:39690
I20260925 21:59:52.511243 19975 tablet_service.cc:1514] Processing DeleteTablet for tablet f53551564691418b8c8e338763903844 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:52 UTC) from {username='slave'} at 127.0.0.1:58380
I20260925 21:59:52.511430 20342 tablet_replica.cc:330] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6: stopping tablet replica
I20260925 21:59:52.511528 20342 raft_consensus.cc:2229] T f53551564691418b8c8e338763903844 P d6d0c3ec959a4614bbce375e624caaa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:59:52.511550 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 19913
I20260925 21:59:52.512290 20340 tablet_replica.cc:330] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a: stopping tablet replica
I20260925 21:59:52.512374 20340 raft_consensus.cc:2229] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:59:52.512476 20340 raft_consensus.cc:2258] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:52.513357 20340 ts_tablet_manager.cc:1905] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:52.515231 20340 ts_tablet_manager.cc:1918] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:52.515290 20340 log.cc:1196] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/f53551564691418b8c8e338763903844
I20260925 21:59:52.515522 20340 ts_tablet_manager.cc:1939] T f53551564691418b8c8e338763903844 P aace8997dc564485a5aee5b0354ac84a: Deleting consensus metadata
I20260925 21:59:52.516388 19838 catalog_manager.cc:4841] TS aace8997dc564485a5aee5b0354ac84a (127.2.70.195:33651): tablet f53551564691418b8c8e338763903844 (table test-table [id=44cbe33f885a43b5a6b98b299b05d901]) successfully deleted
W20260925 21:59:52.516846 19837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:43435 (error 108)
I20260925 21:59:52.516865 20341 tablet_replica.cc:330] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3: stopping tablet replica
I20260925 21:59:52.516985 20341 raft_consensus.cc:2229] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:59:52.517105 20341 raft_consensus.cc:2258] T f53551564691418b8c8e338763903844 P dbfa0447011f411d90458b3301970ea3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:52.517154 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 20044
I20260925 21:59:52.517676 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.517247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:43435 (local address 127.0.0.1:58390)
0925 21:59:52.517603 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:52.517644 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:43435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336}
W20260925 21:59:52.517766 19837 catalog_manager.cc:4584] TS d6d0c3ec959a4614bbce375e624caaa6 (127.2.70.193:43435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f53551564691418b8c8e338763903844: Network error: Client connection negotiation failed: client connection to 127.2.70.193:43435: connect: Connection refused (error 111)
I20260925 21:59:52.522185 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:52.522082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:42499 (local address 127.0.0.1:39704)
0925 21:59:52.522129 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:52.522154 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.194:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 21:59:52.522254 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 20175
I20260925 21:59:52.526585 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 19822
2026-09-25T21:59:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (848 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:59:52 dist-test-slave-2g0f krb5kdc[20346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:59:52 dist-test-slave-2g0f krb5kdc[20346](info): set up 2 sockets
Sep 25 21:59:52 dist-test-slave-2g0f krb5kdc[20346](info): commencing operation
krb5kdc: starting...
W20260925 21:59:53.483433 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:59:53 dist-test-slave-2g0f krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:59:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:59:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:59:53.598701 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:35433
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:35433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:53.679078 20359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:53.679296 20359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:53.679318 20359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:53.680498 20359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:59:53.680534 20359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:53.680546 20359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:53.680558 20359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:59:53.680568 20359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:59:53.682010 20359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:35433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:35433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:53.682217 20359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:53.682403 20359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:53.687269 20365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:53.687269 20367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:53.687388 20364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:53.687399 20359 server_base.cc:1034] running on GCE node
I20260925 21:59:53.687776 20359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:53.688092 20359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:53.689257 20359 hybrid_clock.cc:648] HybridClock initialized: now 1790373593689231 us; error 34 us; skew 500 ppm
Sep 25 21:59:53 dist-test-slave-2g0f krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373593, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:53.691964 20359 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:59:53.692453 20359 webserver.cc:460] Webserver started at http://127.2.70.254:41041/ using document root <none> and password file <none>
I20260925 21:59:53.692685 20359 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:53.692734 20359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:53.692859 20359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:53.693970 20359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "69110776f5ef45048476b08a13f1ce62"
format_stamp: "Formatted at 2026-09-25 21:59:53 on dist-test-slave-2g0f"
server_key: "00f4caff182f343b360c6f102b47b356"
server_key_iv: "3f91f8f25d69d2f8e9346253ccdac649"
server_key_version: "encryptionkey@0"
I20260925 21:59:53.694329 20359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "69110776f5ef45048476b08a13f1ce62"
format_stamp: "Formatted at 2026-09-25 21:59:53 on dist-test-slave-2g0f"
server_key: "00f4caff182f343b360c6f102b47b356"
server_key_iv: "3f91f8f25d69d2f8e9346253ccdac649"
server_key_version: "encryptionkey@0"
I20260925 21:59:53.695781 20359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 21:59:53.696641 20374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:53.696794 20359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:59:53.696890 20359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "69110776f5ef45048476b08a13f1ce62"
format_stamp: "Formatted at 2026-09-25 21:59:53 on dist-test-slave-2g0f"
server_key: "00f4caff182f343b360c6f102b47b356"
server_key_iv: "3f91f8f25d69d2f8e9346253ccdac649"
server_key_version: "encryptionkey@0"
I20260925 21:59:53.697028 20359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:53.720103 20359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:53.722069 20359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:53.722206 20359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:53.726127 20359 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:35433
I20260925 21:59:53.726171 20426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:35433 every 8 connection(s)
I20260925 21:59:53.726519 20359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:59:53.727370 20427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:53.729739 20427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62: Bootstrap starting.
I20260925 21:59:53.730321 20427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:53.730582 20427 log.cc:824] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:53.731391 20427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62: No bootstrap required, opened a new log
I20260925 21:59:53.733055 20427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69110776f5ef45048476b08a13f1ce62" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35433 } }
I20260925 21:59:53.733244 20427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:53.733265 20427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69110776f5ef45048476b08a13f1ce62, State: Initialized, Role: FOLLOWER
I20260925 21:59:53.733402 20427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69110776f5ef45048476b08a13f1ce62" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35433 } }
I20260925 21:59:53.733469 20427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:59:53.733491 20427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:59:53.733538 20427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:53.733783 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 20359
I20260925 21:59:53.733867 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:59:53.734054 2331 external_mini_cluster.cc:1428] Setting key 2adee0d532051e111c26453a016d997c
I20260925 21:59:53.734136 20427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69110776f5ef45048476b08a13f1ce62" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35433 } }
I20260925 21:59:53.734238 20427 leader_election.cc:302] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69110776f5ef45048476b08a13f1ce62; no voters:
I20260925 21:59:53.734411 20427 leader_election.cc:288] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:59:53.734447 20430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:53.734722 20427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:59:53.734792 20430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [term 1 LEADER]: Becoming Leader. State: Replica: 69110776f5ef45048476b08a13f1ce62, State: Running, Role: LEADER
I20260925 21:59:53.734875 20430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69110776f5ef45048476b08a13f1ce62" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35433 } }
I20260925 21:59:53.735378 20434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69110776f5ef45048476b08a13f1ce62. Latest consensus state: current_term: 1 leader_uuid: "69110776f5ef45048476b08a13f1ce62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69110776f5ef45048476b08a13f1ce62" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35433 } } }
I20260925 21:59:53.735613 20434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:53.735532 20432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69110776f5ef45048476b08a13f1ce62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69110776f5ef45048476b08a13f1ce62" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 35433 } } }
I20260925 21:59:53.735862 20432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62 [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:53.736016 20445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:59:53.736598 20445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:59:53.738089 20445 catalog_manager.cc:1347] Generated new cluster ID: 17038c4e0d9e4d1bb7f71b65ab006715
I20260925 21:59:53.738140 20445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 21:59:53 dist-test-slave-2g0f krb5kdc[20346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:53.744916 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:53.734527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.2.70.254:35433)
0925 21:59:53.735242 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:53.735252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:53.735289 (+ 37us) server_negotiation.cc:407] Connection header received
0925 21:59:53.735419 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:53.735427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:53.735492 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:53.735651 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:53.736582 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:53.737694 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:53.738555 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:53.739098 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:53.739223 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:53.741439 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:53.741443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:53.741444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:53.741455 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:53.743424 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:53.743741 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:53.743745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:53.743745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:53.743874 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:53.744057 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:53.744058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:53.744058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:53.744312 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:53.744485 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:53.744554 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 21:59:53.744696 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":113,"threads_started":1}
I20260925 21:59:53.745992 20445 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:59:53.746735 20445 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:59:53.749691 20445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62: Generated new TSK 0
I20260925 21:59:53.749866 20445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:59:53.782850 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35433
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:53.889741 20453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:53.889941 20453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:53.889964 20453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:53.891147 20453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:53.891191 20453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:53.891216 20453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:59:53.892784 20453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:53.893013 20453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:53.893237 20453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:53.898656 20458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:53.898663 20461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:53.898702 20459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:53.898983 20453 server_base.cc:1034] running on GCE node
I20260925 21:59:53.899175 20453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:53.899428 20453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:53.900595 20453 hybrid_clock.cc:648] HybridClock initialized: now 1790373593900570 us; error 33 us; skew 500 ppm
Sep 25 21:59:53 dist-test-slave-2g0f krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373593, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:53.903342 20453 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:59:53.903769 20453 webserver.cc:460] Webserver started at http://127.2.70.193:42159/ using document root <none> and password file <none>
I20260925 21:59:53.904026 20453 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:53.904078 20453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:53.904209 20453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:53.905329 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "863cd4dab5d94061aa0c7f46c26b8755"
format_stamp: "Formatted at 2026-09-25 21:59:53 on dist-test-slave-2g0f"
server_key: "f778f277f6c6a17d922162a03ca396d4"
server_key_iv: "e4b8f3e9a65833513b6e97f24d2a85b1"
server_key_version: "encryptionkey@0"
I20260925 21:59:53.905655 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "863cd4dab5d94061aa0c7f46c26b8755"
format_stamp: "Formatted at 2026-09-25 21:59:53 on dist-test-slave-2g0f"
server_key: "f778f277f6c6a17d922162a03ca396d4"
server_key_iv: "e4b8f3e9a65833513b6e97f24d2a85b1"
server_key_version: "encryptionkey@0"
I20260925 21:59:53.906900 20453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:59:53.907591 20468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:53.907730 20453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:53.907812 20453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "863cd4dab5d94061aa0c7f46c26b8755"
format_stamp: "Formatted at 2026-09-25 21:59:53 on dist-test-slave-2g0f"
server_key: "f778f277f6c6a17d922162a03ca396d4"
server_key_iv: "e4b8f3e9a65833513b6e97f24d2a85b1"
server_key_version: "encryptionkey@0"
I20260925 21:59:53.907920 20453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:53.924224 20453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:53.926081 20453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:53.926221 20453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:53.926437 20453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:53.926761 20453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:53.926801 20453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:53.926831 20453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:53.926853 20453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:53.932350 20453 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:41933
I20260925 21:59:53.932396 20581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:41933 every 8 connection(s)
I20260925 21:59:53.932713 20453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:59:53.938647 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 20453
I20260925 21:59:53.938759 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
Sep 25 21:59:53 dist-test-slave-2g0f krb5kdc[20346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373593, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:53.938995 2331 external_mini_cluster.cc:1428] Setting key dd52d85ddcec8b57b80b488a1689bcfe
I20260925 21:59:53.942080 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:53.933414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:55139 (local address 127.2.70.254:35433)
0925 21:59:53.933468 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:53.933470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:53.933676 (+ 206us) server_negotiation.cc:407] Connection header received
0925 21:59:53.934184 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:53.934186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:53.934196 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:53.934250 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:53.935259 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:53.935879 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:53.936906 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:53.937115 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:53.937179 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:53.939413 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:53.939414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:53.939414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:53.939422 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:53.940784 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:53.941176 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:53.941177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:53.941177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:53.941231 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:53.941459 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:53.941460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:53.941460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:53.941575 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:53.941632 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:53.941898 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 21:59:53.941988 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 21:59:53.942270 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:53.933390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35433 (local address 127.2.70.193:55139)
0925 21:59:53.933550 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:53.933586 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:53.934065 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:53.934315 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:53.934318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:53.934926 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:53.935157 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:53.935162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:53.935968 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:53.935969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:53.936808 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:53.936811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:53.937188 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:53.937188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:53.937293 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:53.937791 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:53.937798 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:53.939315 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:53.940862 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:53.940870 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:53.940873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:53.941108 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:53.941287 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:53.941288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:53.941289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:53.941399 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:53.941630 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:53.941637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:53.941778 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 21:59:53.941867 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 21:59:53.942045 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260925 21:59:53.942725 20582 heartbeater.cc:347] Connected to a master server at 127.2.70.254:35433
I20260925 21:59:53.942836 20582 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:53.943034 20582 heartbeater.cc:511] Master 127.2.70.254:35433 requested a full tablet report, sending...
I20260925 21:59:53.943553 20391 ts_manager.cc:194] Registered new tserver with Master: 863cd4dab5d94061aa0c7f46c26b8755 (127.2.70.193:41933)
I20260925 21:59:53.944219 20391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:55139
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:59:53.964254 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35433
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:54.045186 20587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:54.045415 20587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:54.045439 20587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:54.046666 20587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:54.046713 20587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:54.046741 20587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:59:54.048277 20587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:54.048514 20587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:54.048744 20587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:54.054539 20593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:54.054425 20592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:54.054603 20595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:54.054690 20587 server_base.cc:1034] running on GCE node
I20260925 21:59:54.054899 20587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:54.055192 20587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:54.056354 20587 hybrid_clock.cc:648] HybridClock initialized: now 1790373594056341 us; error 36 us; skew 500 ppm
Sep 25 21:59:54 dist-test-slave-2g0f krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:54.058943 20587 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:59:54.059445 20587 webserver.cc:460] Webserver started at http://127.2.70.194:38953/ using document root <none> and password file <none>
I20260925 21:59:54.059648 20587 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:54.059697 20587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:54.059823 20587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:54.060920 20587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "812c310923fb4503ab468b5ea5fd7911"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "3d18c64f1153b4854ed950cb0c064d0d"
server_key_iv: "5903afffd295409f26eee3fbec97309d"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.061259 20587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "812c310923fb4503ab468b5ea5fd7911"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "3d18c64f1153b4854ed950cb0c064d0d"
server_key_iv: "5903afffd295409f26eee3fbec97309d"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.062497 20587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 21:59:54.063369 20602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.063549 20587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 21:59:54.063634 20587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "812c310923fb4503ab468b5ea5fd7911"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "3d18c64f1153b4854ed950cb0c064d0d"
server_key_iv: "5903afffd295409f26eee3fbec97309d"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.063696 20587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:54.075106 20587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:54.077080 20587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:54.077221 20587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:54.077447 20587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:54.077750 20587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:54.077783 20587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.077812 20587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:54.077836 20587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.083411 20587 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:37191
I20260925 21:59:54.083456 20715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:37191 every 8 connection(s)
I20260925 21:59:54.083763 20587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:59:54.089008 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 20587
I20260925 21:59:54.089124 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:59:54.089347 2331 external_mini_cluster.cc:1428] Setting key 1732ec653b799eaf64f37ae1262c6727
Sep 25 21:59:54 dist-test-slave-2g0f krb5kdc[20346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:54.093915 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.084569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:59619 (local address 127.2.70.254:35433)
0925 21:59:54.084633 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:54.084635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:54.084828 (+ 193us) server_negotiation.cc:407] Connection header received
0925 21:59:54.085421 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:54.085422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:54.085457 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:54.085510 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:54.086484 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.087198 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.088341 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.088555 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.088615 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.091132 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:54.091133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:54.091134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:54.091141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:54.092486 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:54.092868 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:54.092870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:54.092870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:54.092947 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:54.093200 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:54.093201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:54.093201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:54.093332 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:54.093398 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:54.093714 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 21:59:54.093815 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 21:59:54.094161 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.084498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35433 (local address 127.2.70.194:59619)
0925 21:59:54.084688 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:54.084731 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:54.085309 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:54.085594 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:54.085598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:54.086179 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:54.086385 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.086391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.087298 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.087299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.088255 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.088258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.088626 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.088627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.088761 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.089330 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:54.089341 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:54.091036 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:54.092557 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:54.092562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:54.092564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:54.092789 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:54.093010 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:54.093011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:54.093011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:54.093144 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:54.093422 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:54.093429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:54.093565 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 21:59:54.093681 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 21:59:54.093879 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260925 21:59:54.094749 20716 heartbeater.cc:347] Connected to a master server at 127.2.70.254:35433
I20260925 21:59:54.094852 20716 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:54.095141 20716 heartbeater.cc:511] Master 127.2.70.254:35433 requested a full tablet report, sending...
I20260925 21:59:54.095542 20391 ts_manager.cc:194] Registered new tserver with Master: 812c310923fb4503ab468b5ea5fd7911 (127.2.70.194:37191)
I20260925 21:59:54.096082 20391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:59619
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:59:54.114892 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35433
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:54.194382 20721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:54.194600 20721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:54.194624 20721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:54.195788 20721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:54.195856 20721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:54.195883 20721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:59:54.197306 20721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:35433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:54.197495 20721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:54.197672 20721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:54.202720 20729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:54.202713 20726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:54.202932 20721 server_base.cc:1034] running on GCE node
W20260925 21:59:54.202831 20727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:54.203235 20721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:54.203493 20721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:54.204635 20721 hybrid_clock.cc:648] HybridClock initialized: now 1790373594204620 us; error 26 us; skew 500 ppm
Sep 25 21:59:54 dist-test-slave-2g0f krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:54.206979 20721 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:59:54.207412 20721 webserver.cc:460] Webserver started at http://127.2.70.195:38891/ using document root <none> and password file <none>
I20260925 21:59:54.207600 20721 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:54.207638 20721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:54.207729 20721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:54.208801 20721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "165721671d664954be1218b49647ca6a"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "2e83b12b145c8f0b595be7b1b308bcb5"
server_key_iv: "9cc3deddd082529da7d080f852e54a4b"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.209091 20721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "165721671d664954be1218b49647ca6a"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "2e83b12b145c8f0b595be7b1b308bcb5"
server_key_iv: "9cc3deddd082529da7d080f852e54a4b"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.210377 20721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:59:54.211146 20736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.211303 20721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:54.211388 20721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "165721671d664954be1218b49647ca6a"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "2e83b12b145c8f0b595be7b1b308bcb5"
server_key_iv: "9cc3deddd082529da7d080f852e54a4b"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.211459 20721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:54.224848 20721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:54.226783 20721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:54.226898 20721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:54.227097 20721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:54.227371 20721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:54.227401 20721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.227423 20721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:54.227434 20721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.232901 20721 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:44509
I20260925 21:59:54.232987 20849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:44509 every 8 connection(s)
I20260925 21:59:54.233265 20721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
Sep 25 21:59:54 dist-test-slave-2g0f krb5kdc[20346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:54.239918 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 20721
I20260925 21:59:54.240018 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:59:54.240226 2331 external_mini_cluster.cc:1428] Setting key 04a99b013e76a5217371cd9b9922969f
I20260925 21:59:54.242594 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.233969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:35263 (local address 127.2.70.254:35433)
0925 21:59:54.234026 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:54.234028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:54.234238 (+ 210us) server_negotiation.cc:407] Connection header received
0925 21:59:54.234783 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:54.234785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:54.234799 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:54.234851 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:54.235922 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.236497 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.237478 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.237662 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.237707 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.239825 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:54.239862 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:54.239863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:54.239872 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:54.241345 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:54.241711 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:54.241712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:54.241713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:54.241759 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:54.241977 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:54.241978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:54.241979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:54.242103 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:54.242169 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:54.242457 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 21:59:54.242529 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 21:59:54.242751 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.233927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:35433 (local address 127.2.70.195:35263)
0925 21:59:54.234114 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:54.234154 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:54.234684 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:54.234934 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:54.234939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:54.235607 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:54.235804 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.235809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.236571 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.236572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.237401 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.237404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.237718 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.237719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.237814 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.238320 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:54.238328 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:54.239717 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:54.241421 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:54.241426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:54.241428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:54.241656 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:54.241803 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:54.241803 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:54.241803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:54.241894 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:54.242179 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:54.242185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:54.242308 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 21:59:54.242395 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 21:59:54.242523 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260925 21:59:54.243180 20850 heartbeater.cc:347] Connected to a master server at 127.2.70.254:35433
I20260925 21:59:54.243268 20850 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:54.243438 20850 heartbeater.cc:511] Master 127.2.70.254:35433 requested a full tablet report, sending...
I20260925 21:59:54.243788 20391 ts_manager.cc:194] Registered new tserver with Master: 165721671d664954be1218b49647ca6a (127.2.70.195:44509)
I20260925 21:59:54.244197 20391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:35263
I20260925 21:59:54.252230 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:59:54.260210 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.253549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.2.70.254:35433)
0925 21:59:54.253630 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:54.253632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:54.253716 (+ 84us) server_negotiation.cc:407] Connection header received
0925 21:59:54.253797 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:54.253798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:54.253811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:54.253867 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:54.254660 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.255185 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.255987 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.256173 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.256228 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.257645 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:54.257647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:54.257647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:54.257657 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:54.258954 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:54.259394 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:54.259395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:54.259396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:54.259450 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:54.259646 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:54.259646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:54.259647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:54.259808 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:54.259940 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:54.260055 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 21:59:54.260116 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 21:59:54.261905 20391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:59:54.263463 20391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:59:54.269869 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.265956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51130 (local address 127.2.70.193:41933)
0925 21:59:54.266158 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:54.266163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:54.266173 (+ 10us) server_negotiation.cc:407] Connection header received
0925 21:59:54.266216 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:54.266218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:54.266234 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:54.266317 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:54.266577 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.267378 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.268818 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.269486 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.269547 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.269609 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:54.269747 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 21:59:54.269791 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":91,"threads_started":1}
I20260925 21:59:54.269874 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.265824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:41933 (local address 127.0.0.1:51130)
0925 21:59:54.265962 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:54.266009 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:54.266090 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:54.266413 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:54.266416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:54.266418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:54.266490 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.266491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.267499 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.267500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.268721 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.268724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.269556 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.269557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.269644 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.269658 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 21:59:54.269721 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 21:59:54.269781 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260925 21:59:54.270187 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.265824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:44509 (local address 127.0.0.1:37766)
0925 21:59:54.266186 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:54.266194 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:54.266227 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:54.266557 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:54.266558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:54.266560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:54.266616 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.266617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.268025 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.268026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.269144 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.269146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.270000 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.270001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.270070 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.270072 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:59:54.270100 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 21:59:54.270133 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":27,"thread_start_us":72,"threads_started":1}
I20260925 21:59:54.270279 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.266202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.2.70.194:37191)
0925 21:59:54.266383 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:54.266387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:54.266438 (+ 51us) server_negotiation.cc:407] Connection header received
0925 21:59:54.266524 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:54.266526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:54.266542 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:54.266591 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:54.266830 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.267960 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.269014 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.269674 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.269744 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.269797 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:54.270161 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 21:59:54.270207 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260925 21:59:54.270077 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.266039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:37191 (local address 127.0.0.1:54996)
0925 21:59:54.266378 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:54.266389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:54.266451 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:54.266718 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:54.266718 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:54.266720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:54.266754 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.266755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.268057 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.268059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.268939 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.268942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.269833 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.269834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.269899 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.269904 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:59:54.269951 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 21:59:54.269996 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260925 21:59:54.270817 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.266178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.2.70.195:44509)
0925 21:59:54.266391 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:54.266396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:54.266408 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:59:54.266447 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:54.266449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:54.266463 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:54.266690 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:54.266770 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.267921 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.269227 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.269916 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.270603 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.270690 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:54.270710 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 21:59:54.270744 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
I20260925 21:59:54.271057 20516 tablet_service.cc:1467] Processing CreateTablet for tablet 5c90d5b9539f43a69955f7c8974294d2 (DEFAULT_TABLE table=test-table [id=a766e9e6f1004ecaa5b3bbdc227e9782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:54.271190 20650 tablet_service.cc:1467] Processing CreateTablet for tablet 5c90d5b9539f43a69955f7c8974294d2 (DEFAULT_TABLE table=test-table [id=a766e9e6f1004ecaa5b3bbdc227e9782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:54.271443 20650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c90d5b9539f43a69955f7c8974294d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:54.271445 20516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c90d5b9539f43a69955f7c8974294d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:54.271521 20784 tablet_service.cc:1467] Processing CreateTablet for tablet 5c90d5b9539f43a69955f7c8974294d2 (DEFAULT_TABLE table=test-table [id=a766e9e6f1004ecaa5b3bbdc227e9782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:54.271801 20784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c90d5b9539f43a69955f7c8974294d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:54.274034 20867 tablet_bootstrap.cc:492] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911: Bootstrap starting.
I20260925 21:59:54.274578 20867 tablet_bootstrap.cc:654] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:54.274814 20869 tablet_bootstrap.cc:492] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a: Bootstrap starting.
I20260925 21:59:54.275574 20869 tablet_bootstrap.cc:654] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:54.275548 20867 log.cc:824] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:54.275964 20868 tablet_bootstrap.cc:492] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755: Bootstrap starting.
I20260925 21:59:54.276153 20869 log.cc:824] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:54.276458 20867 tablet_bootstrap.cc:492] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911: No bootstrap required, opened a new log
I20260925 21:59:54.276535 20867 ts_tablet_manager.cc:1397] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 21:59:54.276568 20868 tablet_bootstrap.cc:654] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:54.276957 20869 tablet_bootstrap.cc:492] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a: No bootstrap required, opened a new log
I20260925 21:59:54.277025 20869 ts_tablet_manager.cc:1397] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:59:54.277040 20868 log.cc:824] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:54.277654 20868 tablet_bootstrap.cc:492] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755: No bootstrap required, opened a new log
I20260925 21:59:54.277711 20868 ts_tablet_manager.cc:1397] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 21:59:54.278539 20867 raft_consensus.cc:348] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863cd4dab5d94061aa0c7f46c26b8755" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41933 } } peers { permanent_uuid: "812c310923fb4503ab468b5ea5fd7911" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37191 } } peers { permanent_uuid: "165721671d664954be1218b49647ca6a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 44509 } }
I20260925 21:59:54.278728 20867 raft_consensus.cc:374] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:54.278765 20867 raft_consensus.cc:729] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 812c310923fb4503ab468b5ea5fd7911, State: Initialized, Role: FOLLOWER
I20260925 21:59:54.278906 20867 consensus_queue.cc:260] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863cd4dab5d94061aa0c7f46c26b8755" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41933 } } peers { permanent_uuid: "812c310923fb4503ab468b5ea5fd7911" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37191 } } peers { permanent_uuid: "165721671d664954be1218b49647ca6a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 44509 } }
I20260925 21:59:54.279210 20867 ts_tablet_manager.cc:1428] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:59:54.279219 20716 heartbeater.cc:503] Master 127.2.70.254:35433 was elected leader, sending a full tablet report...
I20260925 21:59:54.279260 20868 raft_consensus.cc:348] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863cd4dab5d94061aa0c7f46c26b8755" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41933 } } peers { permanent_uuid: "812c310923fb4503ab468b5ea5fd7911" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37191 } } peers { permanent_uuid: "165721671d664954be1218b49647ca6a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 44509 } }
I20260925 21:59:54.279409 20868 raft_consensus.cc:374] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:54.279439 20868 raft_consensus.cc:729] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863cd4dab5d94061aa0c7f46c26b8755, State: Initialized, Role: FOLLOWER
I20260925 21:59:54.279343 20869 raft_consensus.cc:348] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863cd4dab5d94061aa0c7f46c26b8755" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41933 } } peers { permanent_uuid: "812c310923fb4503ab468b5ea5fd7911" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37191 } } peers { permanent_uuid: "165721671d664954be1218b49647ca6a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 44509 } }
I20260925 21:59:54.279474 20869 raft_consensus.cc:374] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:54.279502 20869 raft_consensus.cc:729] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165721671d664954be1218b49647ca6a, State: Initialized, Role: FOLLOWER
I20260925 21:59:54.279562 20868 consensus_queue.cc:260] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863cd4dab5d94061aa0c7f46c26b8755" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41933 } } peers { permanent_uuid: "812c310923fb4503ab468b5ea5fd7911" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37191 } } peers { permanent_uuid: "165721671d664954be1218b49647ca6a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 44509 } }
I20260925 21:59:54.279606 20869 consensus_queue.cc:260] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863cd4dab5d94061aa0c7f46c26b8755" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41933 } } peers { permanent_uuid: "812c310923fb4503ab468b5ea5fd7911" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37191 } } peers { permanent_uuid: "165721671d664954be1218b49647ca6a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 44509 } }
I20260925 21:59:54.279866 20868 ts_tablet_manager.cc:1428] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 21:59:54.279882 20869 ts_tablet_manager.cc:1428] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:59:54.280009 20850 heartbeater.cc:503] Master 127.2.70.254:35433 was elected leader, sending a full tablet report...
I20260925 21:59:54.280018 20582 heartbeater.cc:503] Master 127.2.70.254:35433 was elected leader, sending a full tablet report...
W20260925 21:59:54.334440 20717 tablet.cc:2369] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 21:59:54.433437 20583 tablet.cc:2369] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:59:54.474386 20874 raft_consensus.cc:482] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:59:54.474527 20874 raft_consensus.cc:504] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863cd4dab5d94061aa0c7f46c26b8755" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41933 } } peers { permanent_uuid: "812c310923fb4503ab468b5ea5fd7911" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37191 } } peers { permanent_uuid: "165721671d664954be1218b49647ca6a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 44509 } }
I20260925 21:59:54.474959 20874 leader_election.cc:288] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 812c310923fb4503ab468b5ea5fd7911 (127.2.70.194:37191), 165721671d664954be1218b49647ca6a (127.2.70.195:44509)
I20260925 21:59:54.477777 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.475225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:57963 (local address 127.2.70.194:37191)
0925 21:59:54.475270 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:54.475272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:54.475342 (+ 70us) server_negotiation.cc:407] Connection header received
0925 21:59:54.475405 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:54.475406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:54.475415 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:54.475455 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:54.475622 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.476120 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.477110 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.477468 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.477529 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.477537 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:54.477675 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 21:59:54.477713 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 21:59:54.477819 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.475229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:60845 (local address 127.2.70.195:44509)
0925 21:59:54.475274 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:54.475276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:54.475322 (+ 46us) server_negotiation.cc:407] Connection header received
0925 21:59:54.475405 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:54.475406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:54.475415 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:54.475461 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:54.475630 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.476130 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.477113 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.477475 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.477542 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.477550 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:54.477702 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 21:59:54.477742 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 21:59:54.477840 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.475130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:44509 (local address 127.2.70.193:60845)
0925 21:59:54.475258 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:54.475270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:54.475336 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:54.475469 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:54.475470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:54.475472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:54.475562 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.475564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.476205 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.476206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.477019 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.477022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.477543 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.477544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.477638 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.477641 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:54.477688 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 21:59:54.477729 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":65,"thread_start_us":58,"threads_started":1}
I20260925 21:59:54.477842 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.475130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:37191 (local address 127.2.70.193:57963)
0925 21:59:54.475265 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:54.475275 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:54.475337 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:54.475464 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:54.475465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:54.475467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:54.475551 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.475553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.476194 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.476195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.477018 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.477020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.477535 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.477536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.477614 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.477617 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:54.477659 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 21:59:54.477701 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":6,"thread_start_us":74,"threads_started":1}
I20260925 21:59:54.478140 20804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c90d5b9539f43a69955f7c8974294d2" candidate_uuid: "863cd4dab5d94061aa0c7f46c26b8755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "165721671d664954be1218b49647ca6a" is_pre_election: true
I20260925 21:59:54.478282 20804 raft_consensus.cc:2454] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 863cd4dab5d94061aa0c7f46c26b8755 in term 0.
I20260925 21:59:54.478353 20670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c90d5b9539f43a69955f7c8974294d2" candidate_uuid: "863cd4dab5d94061aa0c7f46c26b8755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "812c310923fb4503ab468b5ea5fd7911" is_pre_election: true
I20260925 21:59:54.478492 20670 raft_consensus.cc:2454] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 863cd4dab5d94061aa0c7f46c26b8755 in term 0.
I20260925 21:59:54.478509 20470 leader_election.cc:302] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 165721671d664954be1218b49647ca6a, 863cd4dab5d94061aa0c7f46c26b8755; no voters:
I20260925 21:59:54.478654 20874 raft_consensus.cc:2781] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:59:54.478698 20874 raft_consensus.cc:482] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:59:54.478720 20874 raft_consensus.cc:3037] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:54.479400 20874 raft_consensus.cc:504] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863cd4dab5d94061aa0c7f46c26b8755" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41933 } } peers { permanent_uuid: "812c310923fb4503ab468b5ea5fd7911" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37191 } } peers { permanent_uuid: "165721671d664954be1218b49647ca6a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 44509 } }
I20260925 21:59:54.479523 20874 leader_election.cc:288] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [CANDIDATE]: Term 1 election: Requested vote from peers 812c310923fb4503ab468b5ea5fd7911 (127.2.70.194:37191), 165721671d664954be1218b49647ca6a (127.2.70.195:44509)
I20260925 21:59:54.479686 20804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c90d5b9539f43a69955f7c8974294d2" candidate_uuid: "863cd4dab5d94061aa0c7f46c26b8755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "165721671d664954be1218b49647ca6a"
I20260925 21:59:54.479705 20670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c90d5b9539f43a69955f7c8974294d2" candidate_uuid: "863cd4dab5d94061aa0c7f46c26b8755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "812c310923fb4503ab468b5ea5fd7911"
I20260925 21:59:54.479764 20804 raft_consensus.cc:3037] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:54.479777 20670 raft_consensus.cc:3037] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:54.480436 20670 raft_consensus.cc:2454] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 863cd4dab5d94061aa0c7f46c26b8755 in term 1.
I20260925 21:59:54.480522 20804 raft_consensus.cc:2454] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 863cd4dab5d94061aa0c7f46c26b8755 in term 1.
I20260925 21:59:54.480616 20472 leader_election.cc:302] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 812c310923fb4503ab468b5ea5fd7911, 863cd4dab5d94061aa0c7f46c26b8755; no voters:
I20260925 21:59:54.480700 20874 raft_consensus.cc:2781] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:54.480840 20874 raft_consensus.cc:686] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [term 1 LEADER]: Becoming Leader. State: Replica: 863cd4dab5d94061aa0c7f46c26b8755, State: Running, Role: LEADER
I20260925 21:59:54.480935 20874 consensus_queue.cc:237] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863cd4dab5d94061aa0c7f46c26b8755" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41933 } } peers { permanent_uuid: "812c310923fb4503ab468b5ea5fd7911" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37191 } } peers { permanent_uuid: "165721671d664954be1218b49647ca6a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 44509 } }
I20260925 21:59:54.481765 20390 catalog_manager.cc:5495] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 reported cstate change: term changed from 0 to 1, leader changed from <none> to 863cd4dab5d94061aa0c7f46c26b8755 (127.2.70.193). New cstate: current_term: 1 leader_uuid: "863cd4dab5d94061aa0c7f46c26b8755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863cd4dab5d94061aa0c7f46c26b8755" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 41933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "812c310923fb4503ab468b5ea5fd7911" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "165721671d664954be1218b49647ca6a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 44509 } health_report { overall_health: UNKNOWN } } }
W20260925 21:59:54.483765 20851 tablet.cc:2369] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:59:54.501226 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.498683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.2.70.193:41933)
0925 21:59:54.498734 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:54.498736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:54.498745 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:59:54.498799 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:54.498800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:54.498820 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:54.498877 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:59:54.499074 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.499548 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.500436 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.500631 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.500700 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.500872 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:59:54.500959 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:59:54.501027 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:54.501152 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 21:59:54.501162 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 21:59:54.503000 20670 raft_consensus.cc:1261] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911 [term 1 FOLLOWER]: Refusing update from remote peer 863cd4dab5d94061aa0c7f46c26b8755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:54.503005 20804 raft_consensus.cc:1261] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a [term 1 FOLLOWER]: Refusing update from remote peer 863cd4dab5d94061aa0c7f46c26b8755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:54.503322 20874 consensus_queue.cc:1035] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "165721671d664954be1218b49647ca6a" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 44509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:54.503453 20878 consensus_queue.cc:1035] T 5c90d5b9539f43a69955f7c8974294d2 P 863cd4dab5d94061aa0c7f46c26b8755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "812c310923fb4503ab468b5ea5fd7911" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:54.508026 20391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:54.508136 20391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:54.508891 20391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69110776f5ef45048476b08a13f1ce62: Sending DeleteTablet for 3 replicas of tablet 5c90d5b9539f43a69955f7c8974294d2
I20260925 21:59:54.509274 20784 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c90d5b9539f43a69955f7c8974294d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:54 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:37766
I20260925 21:59:54.509305 20516 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c90d5b9539f43a69955f7c8974294d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:54 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:51130
I20260925 21:59:54.509316 20650 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c90d5b9539f43a69955f7c8974294d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:54 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:54996
I20260925 21:59:54.509539 20892 tablet_replica.cc:330] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911: stopping tablet replica
I20260925 21:59:54.509640 20892 raft_consensus.cc:2229] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:59:54.509694 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 20453
I20260925 21:59:54.509817 20892 raft_consensus.cc:2258] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:54.510200 20891 tablet_replica.cc:330] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a: stopping tablet replica
I20260925 21:59:54.510303 20891 raft_consensus.cc:2229] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:59:54.510393 20891 raft_consensus.cc:2258] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:54.510718 20892 ts_tablet_manager.cc:1905] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:54.510747 20891 ts_tablet_manager.cc:1905] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:54.512621 20891 ts_tablet_manager.cc:1918] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:54.512686 20891 log.cc:1196] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/5c90d5b9539f43a69955f7c8974294d2
I20260925 21:59:54.512790 20892 ts_tablet_manager.cc:1918] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:54.512874 20892 log.cc:1196] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/5c90d5b9539f43a69955f7c8974294d2
I20260925 21:59:54.512964 20891 ts_tablet_manager.cc:1939] T 5c90d5b9539f43a69955f7c8974294d2 P 165721671d664954be1218b49647ca6a: Deleting consensus metadata
I20260925 21:59:54.513206 20892 ts_tablet_manager.cc:1939] T 5c90d5b9539f43a69955f7c8974294d2 P 812c310923fb4503ab468b5ea5fd7911: Deleting consensus metadata
I20260925 21:59:54.513345 20376 catalog_manager.cc:4841] TS 165721671d664954be1218b49647ca6a (127.2.70.195:44509): tablet 5c90d5b9539f43a69955f7c8974294d2 (table test-table [id=a766e9e6f1004ecaa5b3bbdc227e9782]) successfully deleted
I20260925 21:59:54.513597 20378 catalog_manager.cc:4841] TS 812c310923fb4503ab468b5ea5fd7911 (127.2.70.194:37191): tablet 5c90d5b9539f43a69955f7c8974294d2 (table test-table [id=a766e9e6f1004ecaa5b3bbdc227e9782]) successfully deleted
W20260925 21:59:54.515853 20378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:41933 (error 108)
I20260925 21:59:54.516009 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 20587
I20260925 21:59:54.516752 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.516633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:41933 (local address 127.0.0.1:51146)
0925 21:59:54.516686 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:54.516722 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 21:59:54.516881 20378 catalog_manager.cc:4584] TS 863cd4dab5d94061aa0c7f46c26b8755 (127.2.70.193:41933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c90d5b9539f43a69955f7c8974294d2: Network error: Client connection negotiation failed: client connection to 127.2.70.193:41933: connect: Connection refused (error 111)
I20260925 21:59:54.521010 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 20721
I20260925 21:59:54.525002 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 20359
2026-09-25T21:59:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1998 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T21:59:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:59:54Z Disabled control of system clock
I20260925 21:59:54.550861 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:39551
--webserver_interface=127.2.70.254
--webserver_port=0
--builtin_ntp_servers=127.2.70.212:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:39551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 21:59:54.631171 20897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:54.631438 20897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:54.631474 20897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:54.632730 20897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:59:54.632776 20897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:54.632792 20897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:54.632810 20897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:59:54.632825 20897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:59:54.634239 20897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:39551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:39551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:54.634505 20897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:54.634800 20897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:54.639497 20905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:54.639549 20903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:54.639500 20902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:54.639751 20897 server_base.cc:1034] running on GCE node
I20260925 21:59:54.640010 20897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:54.640282 20897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:54.641438 20897 hybrid_clock.cc:648] HybridClock initialized: now 1790373594641414 us; error 35 us; skew 500 ppm
I20260925 21:59:54.642666 20897 webserver.cc:460] Webserver started at http://127.2.70.254:36625/ using document root <none> and password file <none>
I20260925 21:59:54.642913 20897 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:54.642966 20897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:54.643090 20897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:54.644213 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "0f1db8cde12040af94be1fe96e22321a"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "c00b89381e5f89e90072948478daa987"
server_key_iv: "be154a00ae242564216be1816eef3a2c"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.644584 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "0f1db8cde12040af94be1fe96e22321a"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "c00b89381e5f89e90072948478daa987"
server_key_iv: "be154a00ae242564216be1816eef3a2c"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.645789 20897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:59:54.646557 20911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.646747 20897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:54.646831 20897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "0f1db8cde12040af94be1fe96e22321a"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "c00b89381e5f89e90072948478daa987"
server_key_iv: "be154a00ae242564216be1816eef3a2c"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.646903 20897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:54.659606 20897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:54.659931 20897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:54.660065 20897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:54.663640 20897 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:39551
I20260925 21:59:54.663693 20963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:39551 every 8 connection(s)
I20260925 21:59:54.664049 20897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:59:54.664978 20964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:54.665230 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 20897
I20260925 21:59:54.665333 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:59:54.665573 2331 external_mini_cluster.cc:1428] Setting key ea21a3123475a3c32a58beae52f083ad
I20260925 21:59:54.667694 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a: Bootstrap starting.
I20260925 21:59:54.668301 20964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:54.668565 20964 log.cc:824] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:54.669204 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a: No bootstrap required, opened a new log
I20260925 21:59:54.670764 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.666117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43916 (local address 127.2.70.254:39551)
0925 21:59:54.666311 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:54.666318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:54.666342 (+ 24us) server_negotiation.cc:407] Connection header received
0925 21:59:54.666770 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:54.666776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:54.666836 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:54.666973 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:54.667172 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.668105 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.668943 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.669368 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.669472 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.670191 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:54.670195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:54.670195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:54.670344 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:54.670375 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:54.670386 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:54.670437 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:54.670526 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 21:59:54.670603 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260925 21:59:54.671221 20964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f1db8cde12040af94be1fe96e22321a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39551 } }
I20260925 21:59:54.671387 20964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:54.671445 20964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f1db8cde12040af94be1fe96e22321a, State: Initialized, Role: FOLLOWER
I20260925 21:59:54.671543 20964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f1db8cde12040af94be1fe96e22321a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39551 } }
I20260925 21:59:54.671603 20964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:59:54.671633 20964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:59:54.671669 20964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:54.672235 20964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f1db8cde12040af94be1fe96e22321a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39551 } }
I20260925 21:59:54.672327 20964 leader_election.cc:302] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f1db8cde12040af94be1fe96e22321a; no voters:
I20260925 21:59:54.672468 20964 leader_election.cc:288] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:59:54.672511 20969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:54.672645 20969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [term 1 LEADER]: Becoming Leader. State: Replica: 0f1db8cde12040af94be1fe96e22321a, State: Running, Role: LEADER
I20260925 21:59:54.672737 20964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:59:54.672771 20969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f1db8cde12040af94be1fe96e22321a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39551 } }
I20260925 21:59:54.673173 20971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f1db8cde12040af94be1fe96e22321a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f1db8cde12040af94be1fe96e22321a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39551 } } }
I20260925 21:59:54.673182 20970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f1db8cde12040af94be1fe96e22321a. Latest consensus state: current_term: 1 leader_uuid: "0f1db8cde12040af94be1fe96e22321a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f1db8cde12040af94be1fe96e22321a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39551 } } }
I20260925 21:59:54.673288 20971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:54.673293 20970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:54.673697 20978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:59:54.674185 20978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:59:54.675678 20978 catalog_manager.cc:1347] Generated new cluster ID: 8d099a2fafde4ac69ed61c5a11f2f3d8
I20260925 21:59:54.675731 20978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:59:54.699410 20978 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:59:54.700006 20978 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:59:54.703708 20978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a: Generated new TSK 0
I20260925 21:59:54.703898 20978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:59:54.710740 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39551
--builtin_ntp_servers=127.2.70.212:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 21:59:54.791819 20988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:54.792080 20988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:54.792104 20988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:54.793298 20988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:54.793339 20988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:54.793363 20988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:59:54.794729 20988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:54.794989 20988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:54.795277 20988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:54.800279 20993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:54.800279 20994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:54.800288 20996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:54.800684 20988 server_base.cc:1034] running on GCE node
I20260925 21:59:54.800899 20988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:54.801170 20988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:54.802325 20988 hybrid_clock.cc:648] HybridClock initialized: now 1790373594802309 us; error 30 us; skew 500 ppm
I20260925 21:59:54.803613 20988 webserver.cc:460] Webserver started at http://127.2.70.193:41811/ using document root <none> and password file <none>
I20260925 21:59:54.803900 20988 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:54.803957 20988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:54.804086 20988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:54.805198 20988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "5940d3aabce24248a3a1daf4cd4a2e99"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "4aba966faa86d13c40e7d48b9821e510"
server_key_iv: "64bbbb346a74333cbb94a11e41624d96"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.805542 20988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "5940d3aabce24248a3a1daf4cd4a2e99"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "4aba966faa86d13c40e7d48b9821e510"
server_key_iv: "64bbbb346a74333cbb94a11e41624d96"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.806824 20988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:59:54.807621 21002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.807761 20988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:54.807863 20988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "5940d3aabce24248a3a1daf4cd4a2e99"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "4aba966faa86d13c40e7d48b9821e510"
server_key_iv: "64bbbb346a74333cbb94a11e41624d96"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.807937 20988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:54.818116 20988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:54.818400 20988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:54.818526 20988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:54.818724 20988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:54.819031 20988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:54.819064 20988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.819092 20988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:54.819118 20988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.824544 20988 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:40299
I20260925 21:59:54.824611 21115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:40299 every 8 connection(s)
I20260925 21:59:54.824911 20988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:59:54.825350 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 20988
I20260925 21:59:54.825438 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:59:54.825645 2331 external_mini_cluster.cc:1428] Setting key 6090bc4580acfb166acdfea1b20bcf3a
I20260925 21:59:54.826563 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39551
--builtin_ntp_servers=127.2.70.212:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 21:59:54.830405 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.825669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:48045 (local address 127.2.70.254:39551)
0925 21:59:54.825722 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:54.825724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:54.826010 (+ 286us) server_negotiation.cc:407] Connection header received
0925 21:59:54.826498 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:54.826500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:54.826514 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:54.826565 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:54.827258 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.828001 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.829005 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.829234 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.829299 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.830139 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:54.830140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:54.830141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:54.830145 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:54.830161 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:54.830164 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:54.830209 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:54.830287 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 21:59:54.830324 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 21:59:54.830626 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.825658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:39551 (local address 127.2.70.193:48045)
0925 21:59:54.825852 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:54.825892 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:54.826401 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:54.826637 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:54.826642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:54.826654 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:54.827173 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.827178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.828105 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.828106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.828902 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.828905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.829320 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.829321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.829467 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.829997 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:59:54.829997 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:54.830026 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:59:54.830050 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:54.830216 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:54.830220 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:54.830223 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:54.830283 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 21:59:54.830390 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260925 21:59:54.831101 21116 heartbeater.cc:347] Connected to a master server at 127.2.70.254:39551
I20260925 21:59:54.831189 21116 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:54.831357 21116 heartbeater.cc:511] Master 127.2.70.254:39551 requested a full tablet report, sending...
I20260925 21:59:54.831951 20928 ts_manager.cc:194] Registered new tserver with Master: 5940d3aabce24248a3a1daf4cd4a2e99 (127.2.70.193:40299)
I20260925 21:59:54.832875 20928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.193:48045
W20260925 21:59:54.909288 21119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:54.909511 21119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:54.909543 21119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:54.910817 21119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:54.910857 21119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:54.910890 21119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:59:54.912518 21119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:54.912760 21119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:54.913050 21119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:54.917929 21127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:54.918093 21119 server_base.cc:1034] running on GCE node
W20260925 21:59:54.917946 21124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:54.917984 21125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:54.918432 21119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:54.918660 21119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:54.919797 21119 hybrid_clock.cc:648] HybridClock initialized: now 1790373594919786 us; error 28 us; skew 500 ppm
I20260925 21:59:54.921029 21119 webserver.cc:460] Webserver started at http://127.2.70.194:36089/ using document root <none> and password file <none>
I20260925 21:59:54.921273 21119 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:54.921319 21119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:54.921437 21119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:54.922489 21119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "56923b764ffc4cbdbcc391b4186bcf2a"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "87a19dd81e329df8cd5c0c6dbbc73b1d"
server_key_iv: "5171a036450531430664b31330962efa"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.922818 21119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "56923b764ffc4cbdbcc391b4186bcf2a"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "87a19dd81e329df8cd5c0c6dbbc73b1d"
server_key_iv: "5171a036450531430664b31330962efa"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.924160 21119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:59:54.924875 21133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.925017 21119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.925091 21119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "56923b764ffc4cbdbcc391b4186bcf2a"
format_stamp: "Formatted at 2026-09-25 21:59:54 on dist-test-slave-2g0f"
server_key: "87a19dd81e329df8cd5c0c6dbbc73b1d"
server_key_iv: "5171a036450531430664b31330962efa"
server_key_version: "encryptionkey@0"
I20260925 21:59:54.925155 21119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:54.942840 21119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:54.943145 21119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:54.943269 21119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:54.943480 21119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:54.943802 21119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:54.943886 21119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.943925 21119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:54.943949 21119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:54.949023 21119 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:41711
I20260925 21:59:54.949100 21246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:41711 every 8 connection(s)
I20260925 21:59:54.949397 21119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:59:54.952109 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 21119
I20260925 21:59:54.952186 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:59:54.952378 2331 external_mini_cluster.cc:1428] Setting key ad8bb7f23418b7d2e776264791ed1137
I20260925 21:59:54.953359 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39551
--builtin_ntp_servers=127.2.70.212:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 21:59:54.954902 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.950044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:53439 (local address 127.2.70.254:39551)
0925 21:59:54.950087 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:54.950090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:54.950355 (+ 265us) server_negotiation.cc:407] Connection header received
0925 21:59:54.950900 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:54.950902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:54.950914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:54.950968 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:54.951631 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.952160 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.953476 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.953733 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.953787 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.954634 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:54.954636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:54.954636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:54.954641 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:54.954659 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:54.954664 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:54.954718 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:54.954790 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 21:59:54.954822 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 21:59:54.955047 21249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:54.950045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:39551 (local address 127.2.70.194:53439)
0925 21:59:54.950217 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:54.950260 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:54.950799 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:54.951044 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:54.951049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:54.951059 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:54.951545 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.951550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.952263 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.952264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.953392 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:54.953395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:54.953814 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:54.953815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:54.953965 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:54.954459 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:59:54.954459 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:54.954503 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:59:54.954535 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:54.954726 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:54.954730 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:54.954732 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:59:54.954782 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 21:59:54.954861 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260925 21:59:54.955449 21247 heartbeater.cc:347] Connected to a master server at 127.2.70.254:39551
I20260925 21:59:54.955534 21247 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:54.955703 21247 heartbeater.cc:511] Master 127.2.70.254:39551 requested a full tablet report, sending...
I20260925 21:59:54.956096 20928 ts_manager.cc:194] Registered new tserver with Master: 56923b764ffc4cbdbcc391b4186bcf2a (127.2.70.194:41711)
I20260925 21:59:54.956449 20928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.194:53439
W20260925 21:59:55.032776 21250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:55.032987 21250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:55.033025 21250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:55.034168 21250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:55.034205 21250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:55.034238 21250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:59:55.035552 21250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:55.035781 21250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:55.036085 21250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:55.040784 21258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:55.040825 21255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:55.040957 21250 server_base.cc:1034] running on GCE node
W20260925 21:59:55.040789 21256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:55.041285 21250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:55.041503 21250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:55.042655 21250 hybrid_clock.cc:648] HybridClock initialized: now 1790373595042644 us; error 40 us; skew 500 ppm
I20260925 21:59:55.044044 21250 webserver.cc:460] Webserver started at http://127.2.70.195:36379/ using document root <none> and password file <none>
I20260925 21:59:55.044269 21250 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:55.044338 21250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:55.044446 21250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:55.045485 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "bed8232116244564a8728ccf7f4a06ac"
format_stamp: "Formatted at 2026-09-25 21:59:55 on dist-test-slave-2g0f"
server_key: "cb70c4e7a90e5dc6414bdf6152142ec2"
server_key_iv: "b5123998897614ccb88768cc4fe96672"
server_key_version: "encryptionkey@0"
I20260925 21:59:55.045775 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "bed8232116244564a8728ccf7f4a06ac"
format_stamp: "Formatted at 2026-09-25 21:59:55 on dist-test-slave-2g0f"
server_key: "cb70c4e7a90e5dc6414bdf6152142ec2"
server_key_iv: "b5123998897614ccb88768cc4fe96672"
server_key_version: "encryptionkey@0"
I20260925 21:59:55.046919 21250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:59:55.047556 21264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:55.047717 21250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:55.047796 21250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "bed8232116244564a8728ccf7f4a06ac"
format_stamp: "Formatted at 2026-09-25 21:59:55 on dist-test-slave-2g0f"
server_key: "cb70c4e7a90e5dc6414bdf6152142ec2"
server_key_iv: "b5123998897614ccb88768cc4fe96672"
server_key_version: "encryptionkey@0"
I20260925 21:59:55.047883 21250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:55.059250 21250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:55.059556 21250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:55.059679 21250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:55.059933 21250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:55.060273 21250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:55.060308 21250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:55.060338 21250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:55.060361 21250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:55.065713 21250 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:42275
I20260925 21:59:55.065768 21377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:42275 every 8 connection(s)
I20260925 21:59:55.066087 21250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:59:55.068275 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 21250
I20260925 21:59:55.068379 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:59:55.068580 2331 external_mini_cluster.cc:1428] Setting key e15aeecd832477ec6b61f54b783e04e8
I20260925 21:59:55.071202 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.066805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:42255 (local address 127.2.70.254:39551)
0925 21:59:55.066870 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:55.066872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:55.067101 (+ 229us) server_negotiation.cc:407] Connection header received
0925 21:59:55.067601 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:55.067602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:55.067611 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:55.067649 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:55.068397 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.069013 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.070075 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.070263 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.070314 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.071009 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:55.071011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:55.071011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:55.071015 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:55.071028 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:55.071031 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:55.071069 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:55.071125 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 21:59:55.071144 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 21:59:55.071389 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.066779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:39551 (local address 127.2.70.195:42255)
0925 21:59:55.066988 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:55.067022 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:55.067511 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:55.067701 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:55.067704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:55.067713 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:55.068301 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:55.068308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.069108 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.069109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:55.070009 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:55.070011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.070320 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.070321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:55.070422 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.070890 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:59:55.070891 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:55.070918 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:59:55.070941 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:55.071074 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:55.071076 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:55.071078 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:59:55.071122 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 21:59:55.071197 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260925 21:59:55.071818 21378 heartbeater.cc:347] Connected to a master server at 127.2.70.254:39551
I20260925 21:59:55.071928 21378 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:55.072116 21378 heartbeater.cc:511] Master 127.2.70.254:39551 requested a full tablet report, sending...
I20260925 21:59:55.072451 20928 ts_manager.cc:194] Registered new tserver with Master: bed8232116244564a8728ccf7f4a06ac (127.2.70.195:42275)
I20260925 21:59:55.072795 20928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.195:42255
I20260925 21:59:55.080431 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:59:55.084296 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.081710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.2.70.254:39551)
0925 21:59:55.081755 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:55.081757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:55.081877 (+ 120us) server_negotiation.cc:407] Connection header received
0925 21:59:55.081921 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:55.081922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:55.081931 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:55.081973 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:55.082083 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.082553 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.083258 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.083444 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.083492 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.084061 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:55.084062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:55.084063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:55.084067 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:55.084081 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:55.084084 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:55.084157 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:55.084209 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 21:59:55.084218 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 21:59:55.085649 20928 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:59:55.087141 20928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:59:55.093497 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.089498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:40299 (local address 127.0.0.1:39220)
0925 21:59:55.089857 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:55.089868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:55.089928 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:55.090212 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:55.090215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:55.090217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:55.090283 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:55.090285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.091145 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.091146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:55.092256 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:55.092258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.093192 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.093193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:55.093274 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.093285 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 21:59:55.093361 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 21:59:55.093413 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":31,"thread_start_us":70,"threads_started":1}
I20260925 21:59:55.093510 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.089603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.2.70.193:40299)
0925 21:59:55.089847 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:55.089852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:55.089998 (+ 146us) server_negotiation.cc:407] Connection header received
0925 21:59:55.090044 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:55.090047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:55.090063 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:55.090138 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:55.090343 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.091063 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.092340 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.093071 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.093195 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.093307 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:55.093375 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 21:59:55.093427 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
I20260925 21:59:55.093889 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.089498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:41711 (local address 127.0.0.1:46096)
0925 21:59:55.089815 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:55.089854 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:55.089920 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:55.090224 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:55.090225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:55.090226 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:55.090287 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:55.090289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.091644 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.091645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:55.092596 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:55.092598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.093617 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.093619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:55.093686 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.093691 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:59:55.093767 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 21:59:55.093811 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":28,"thread_start_us":77,"threads_started":1}
I20260925 21:59:55.093915 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.089603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.2.70.194:41711)
0925 21:59:55.089797 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:55.089801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:55.090000 (+ 199us) server_negotiation.cc:407] Connection header received
0925 21:59:55.090045 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:55.090048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:55.090064 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:55.090131 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:55.090371 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.091553 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.092671 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.093304 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.093372 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.093564 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:55.093787 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 21:59:55.093830 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260925 21:59:55.094237 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.089701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:42275 (local address 127.0.0.1:39458)
0925 21:59:55.090022 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:55.090031 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:55.090070 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:55.090435 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:55.090435 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:55.090437 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:55.090467 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:55.090468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.091674 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.091675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:55.092813 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:55.092815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.093992 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.093993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:55.094072 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.094077 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:59:55.094132 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 21:59:55.094168 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":45,"thread_start_us":56,"threads_started":1}
I20260925 21:59:55.094597 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.089805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.2.70.195:42275)
0925 21:59:55.090236 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:55.090239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:55.090248 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:59:55.090282 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:55.090283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:55.090296 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:55.090361 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:55.090536 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.091613 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.092898 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.093722 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.093796 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.093950 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:55.094467 (+ 517us) server_negotiation.cc:299] Negotiation successful
0925 21:59:55.094512 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":80,"threads_started":1}
I20260925 21:59:55.094620 21050 tablet_service.cc:1467] Processing CreateTablet for tablet f9a137c1e7004887b2ff209a765e208b (DEFAULT_TABLE table=test-table [id=7f8ec1d5c3a244d49a5821c3945d4e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:55.094997 21050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a137c1e7004887b2ff209a765e208b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:55.094913 21181 tablet_service.cc:1467] Processing CreateTablet for tablet f9a137c1e7004887b2ff209a765e208b (DEFAULT_TABLE table=test-table [id=7f8ec1d5c3a244d49a5821c3945d4e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:55.095176 21312 tablet_service.cc:1467] Processing CreateTablet for tablet f9a137c1e7004887b2ff209a765e208b (DEFAULT_TABLE table=test-table [id=7f8ec1d5c3a244d49a5821c3945d4e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:55.095312 21181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a137c1e7004887b2ff209a765e208b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:55.095484 21312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a137c1e7004887b2ff209a765e208b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:55.098251 21392 tablet_bootstrap.cc:492] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac: Bootstrap starting.
I20260925 21:59:55.098603 21393 tablet_bootstrap.cc:492] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a: Bootstrap starting.
I20260925 21:59:55.098989 21392 tablet_bootstrap.cc:654] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:55.099193 21393 tablet_bootstrap.cc:654] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:55.099290 21392 log.cc:824] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:55.099493 21393 log.cc:824] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:55.099795 21394 tablet_bootstrap.cc:492] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99: Bootstrap starting.
I20260925 21:59:55.100088 21392 tablet_bootstrap.cc:492] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac: No bootstrap required, opened a new log
I20260925 21:59:55.100139 21393 tablet_bootstrap.cc:492] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a: No bootstrap required, opened a new log
I20260925 21:59:55.100163 21392 ts_tablet_manager.cc:1397] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:59:55.100196 21393 ts_tablet_manager.cc:1397] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:59:55.100528 21394 tablet_bootstrap.cc:654] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:55.100896 21394 log.cc:824] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:55.101665 21394 tablet_bootstrap.cc:492] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99: No bootstrap required, opened a new log
I20260925 21:59:55.101742 21394 ts_tablet_manager.cc:1397] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:59:55.102263 21393 raft_consensus.cc:348] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41711 } } peers { permanent_uuid: "bed8232116244564a8728ccf7f4a06ac" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 42275 } } peers { permanent_uuid: "5940d3aabce24248a3a1daf4cd4a2e99" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40299 } }
I20260925 21:59:55.102396 21393 raft_consensus.cc:374] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:55.102428 21393 raft_consensus.cc:729] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56923b764ffc4cbdbcc391b4186bcf2a, State: Initialized, Role: FOLLOWER
I20260925 21:59:55.102577 21393 consensus_queue.cc:260] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41711 } } peers { permanent_uuid: "bed8232116244564a8728ccf7f4a06ac" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 42275 } } peers { permanent_uuid: "5940d3aabce24248a3a1daf4cd4a2e99" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40299 } }
I20260925 21:59:55.102823 21393 ts_tablet_manager.cc:1428] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:59:55.102715 21392 raft_consensus.cc:348] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41711 } } peers { permanent_uuid: "bed8232116244564a8728ccf7f4a06ac" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 42275 } } peers { permanent_uuid: "5940d3aabce24248a3a1daf4cd4a2e99" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40299 } }
I20260925 21:59:55.102859 21392 raft_consensus.cc:374] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:55.102886 21392 raft_consensus.cc:729] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed8232116244564a8728ccf7f4a06ac, State: Initialized, Role: FOLLOWER
I20260925 21:59:55.103005 21392 consensus_queue.cc:260] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41711 } } peers { permanent_uuid: "bed8232116244564a8728ccf7f4a06ac" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 42275 } } peers { permanent_uuid: "5940d3aabce24248a3a1daf4cd4a2e99" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40299 } }
I20260925 21:59:55.103044 21247 heartbeater.cc:503] Master 127.2.70.254:39551 was elected leader, sending a full tablet report...
I20260925 21:59:55.103241 21392 ts_tablet_manager.cc:1428] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:59:55.103283 21378 heartbeater.cc:503] Master 127.2.70.254:39551 was elected leader, sending a full tablet report...
I20260925 21:59:55.104198 21394 raft_consensus.cc:348] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41711 } } peers { permanent_uuid: "bed8232116244564a8728ccf7f4a06ac" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 42275 } } peers { permanent_uuid: "5940d3aabce24248a3a1daf4cd4a2e99" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40299 } }
I20260925 21:59:55.104328 21394 raft_consensus.cc:374] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:55.104348 21394 raft_consensus.cc:729] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5940d3aabce24248a3a1daf4cd4a2e99, State: Initialized, Role: FOLLOWER
I20260925 21:59:55.104456 21394 consensus_queue.cc:260] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41711 } } peers { permanent_uuid: "bed8232116244564a8728ccf7f4a06ac" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 42275 } } peers { permanent_uuid: "5940d3aabce24248a3a1daf4cd4a2e99" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40299 } }
I20260925 21:59:55.104663 21394 ts_tablet_manager.cc:1428] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:59:55.104709 21116 heartbeater.cc:503] Master 127.2.70.254:39551 was elected leader, sending a full tablet report...
I20260925 21:59:55.150146 21398 raft_consensus.cc:482] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:59:55.150338 21398 raft_consensus.cc:504] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41711 } } peers { permanent_uuid: "bed8232116244564a8728ccf7f4a06ac" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 42275 } } peers { permanent_uuid: "5940d3aabce24248a3a1daf4cd4a2e99" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40299 } }
I20260925 21:59:55.150790 21398 leader_election.cc:288] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bed8232116244564a8728ccf7f4a06ac (127.2.70.195:42275), 5940d3aabce24248a3a1daf4cd4a2e99 (127.2.70.193:40299)
I20260925 21:59:55.153733 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.151045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:39363 (local address 127.2.70.193:40299)
0925 21:59:55.151105 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:55.151111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:55.151204 (+ 93us) server_negotiation.cc:407] Connection header received
0925 21:59:55.151280 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:55.151281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:55.151292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:55.151350 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:55.151534 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.152035 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.153016 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.153391 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.153452 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.153459 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:55.153622 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 21:59:55.153660 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 21:59:55.153735 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.150953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:40299 (local address 127.2.70.194:39363)
0925 21:59:55.151126 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:55.151136 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:55.151215 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:55.151396 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:55.151397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:55.151398 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:55.151465 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:55.151466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.152111 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.152112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:55.152920 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:55.152931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.153479 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.153480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:55.153555 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.153560 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 21:59:55.153610 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 21:59:55.153649 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":12,"thread_start_us":86,"threads_started":1}
I20260925 21:59:55.153818 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.151080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:34255 (local address 127.2.70.195:42275)
0925 21:59:55.151158 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:55.151160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:55.151169 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:59:55.151266 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:55.151268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:55.151278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:55.151338 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:55.151500 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.152042 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.153057 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.153487 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.153560 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.153567 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:55.153719 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 21:59:55.153759 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 21:59:55.153854 21249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.150953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:42275 (local address 127.2.70.194:34255)
0925 21:59:55.151096 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:55.151109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:55.151180 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:55.151333 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:55.151334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:55.151336 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:55.151433 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:55.151439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.152117 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.152118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:55.152955 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:55.152958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.153578 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.153579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:55.153644 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.153647 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:55.153709 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 21:59:55.153748 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":36}
I20260925 21:59:55.154151 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a137c1e7004887b2ff209a765e208b" candidate_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed8232116244564a8728ccf7f4a06ac" is_pre_election: true
I20260925 21:59:55.154150 21070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a137c1e7004887b2ff209a765e208b" candidate_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5940d3aabce24248a3a1daf4cd4a2e99" is_pre_election: true
I20260925 21:59:55.154282 21332 raft_consensus.cc:2454] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56923b764ffc4cbdbcc391b4186bcf2a in term 0.
I20260925 21:59:55.154289 21070 raft_consensus.cc:2454] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56923b764ffc4cbdbcc391b4186bcf2a in term 0.
I20260925 21:59:55.154505 21134 leader_election.cc:302] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56923b764ffc4cbdbcc391b4186bcf2a, 5940d3aabce24248a3a1daf4cd4a2e99; no voters:
I20260925 21:59:55.154624 21398 raft_consensus.cc:2781] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:59:55.154696 21398 raft_consensus.cc:482] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:59:55.154723 21398 raft_consensus.cc:3037] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:55.155449 21398 raft_consensus.cc:504] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41711 } } peers { permanent_uuid: "bed8232116244564a8728ccf7f4a06ac" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 42275 } } peers { permanent_uuid: "5940d3aabce24248a3a1daf4cd4a2e99" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40299 } }
I20260925 21:59:55.155596 21398 leader_election.cc:288] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [CANDIDATE]: Term 1 election: Requested vote from peers bed8232116244564a8728ccf7f4a06ac (127.2.70.195:42275), 5940d3aabce24248a3a1daf4cd4a2e99 (127.2.70.193:40299)
I20260925 21:59:55.155746 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a137c1e7004887b2ff209a765e208b" candidate_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed8232116244564a8728ccf7f4a06ac"
I20260925 21:59:55.155766 21070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a137c1e7004887b2ff209a765e208b" candidate_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5940d3aabce24248a3a1daf4cd4a2e99"
I20260925 21:59:55.155853 21332 raft_consensus.cc:3037] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:55.155856 21070 raft_consensus.cc:3037] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:55.156569 21070 raft_consensus.cc:2454] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56923b764ffc4cbdbcc391b4186bcf2a in term 1.
I20260925 21:59:55.156631 21332 raft_consensus.cc:2454] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56923b764ffc4cbdbcc391b4186bcf2a in term 1.
I20260925 21:59:55.156755 21134 leader_election.cc:302] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56923b764ffc4cbdbcc391b4186bcf2a, 5940d3aabce24248a3a1daf4cd4a2e99; no voters:
I20260925 21:59:55.156862 21398 raft_consensus.cc:2781] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:55.157065 21398 raft_consensus.cc:686] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 1 LEADER]: Becoming Leader. State: Replica: 56923b764ffc4cbdbcc391b4186bcf2a, State: Running, Role: LEADER
I20260925 21:59:55.157152 21398 consensus_queue.cc:237] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41711 } } peers { permanent_uuid: "bed8232116244564a8728ccf7f4a06ac" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 42275 } } peers { permanent_uuid: "5940d3aabce24248a3a1daf4cd4a2e99" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40299 } }
I20260925 21:59:55.158051 20926 catalog_manager.cc:5495] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 56923b764ffc4cbdbcc391b4186bcf2a (127.2.70.194). New cstate: current_term: 1 leader_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56923b764ffc4cbdbcc391b4186bcf2a" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 41711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed8232116244564a8728ccf7f4a06ac" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 42275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5940d3aabce24248a3a1daf4cd4a2e99" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40299 } health_report { overall_health: UNKNOWN } } }
I20260925 21:59:55.166299 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.164270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.2.70.194:41711)
0925 21:59:55.164346 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:55.164348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:55.164357 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:59:55.164383 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:55.164384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:55.164399 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:55.164459 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:59:55.164551 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.164998 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.165672 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:55.165843 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:55.165884 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:55.166021 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:59:55.166101 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:59:55.166170 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:55.166206 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 21:59:55.166212 (+ 6us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 21:59:55.167919 21332 raft_consensus.cc:1261] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac [term 1 FOLLOWER]: Refusing update from remote peer 56923b764ffc4cbdbcc391b4186bcf2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:55.167932 21070 raft_consensus.cc:1261] T f9a137c1e7004887b2ff209a765e208b P 5940d3aabce24248a3a1daf4cd4a2e99 [term 1 FOLLOWER]: Refusing update from remote peer 56923b764ffc4cbdbcc391b4186bcf2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:55.168258 21402 consensus_queue.cc:1035] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5940d3aabce24248a3a1daf4cd4a2e99" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:55.168375 21398 consensus_queue.cc:1035] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed8232116244564a8728ccf7f4a06ac" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 42275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:55.170706 21406 mvcc.cc:205] Tried to move back new op lower bound from 7333370245805309952 to 7333370245764317184. Current Snapshot: MvccSnapshot[applied={T|T < 7333370245805309952}]
I20260925 21:59:55.171047 21407 mvcc.cc:205] Tried to move back new op lower bound from 7333370245805309952 to 7333370245764317184. Current Snapshot: MvccSnapshot[applied={T|T < 7333370245805309952 or (T in {7333370245805309952})}]
I20260925 21:59:55.172807 20926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:55.172950 20926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:55.173702 20926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f1db8cde12040af94be1fe96e22321a: Sending DeleteTablet for 3 replicas of tablet f9a137c1e7004887b2ff209a765e208b
I20260925 21:59:55.173987 21181 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a137c1e7004887b2ff209a765e208b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:55 UTC) from {username='slave'} at 127.0.0.1:46096
I20260925 21:59:55.174049 21312 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a137c1e7004887b2ff209a765e208b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:55 UTC) from {username='slave'} at 127.0.0.1:39458
I20260925 21:59:55.174070 21050 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a137c1e7004887b2ff209a765e208b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:55 UTC) from {username='slave'} at 127.0.0.1:39220
I20260925 21:59:55.174350 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 20988
I20260925 21:59:55.175505 21416 tablet_replica.cc:330] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac: stopping tablet replica
I20260925 21:59:55.175637 21416 raft_consensus.cc:2229] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:59:55.175788 21416 raft_consensus.cc:2258] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:55.175997 21415 tablet_replica.cc:330] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a: stopping tablet replica
I20260925 21:59:55.176358 21416 ts_tablet_manager.cc:1905] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:55.177683 21415 raft_consensus.cc:2229] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:59:55.177812 21415 raft_consensus.cc:2258] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:55.178095 21416 ts_tablet_manager.cc:1918] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:55.178198 21416 log.cc:1196] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/f9a137c1e7004887b2ff209a765e208b
I20260925 21:59:55.178226 21415 ts_tablet_manager.cc:1905] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:55.178496 21416 ts_tablet_manager.cc:1939] T f9a137c1e7004887b2ff209a765e208b P bed8232116244564a8728ccf7f4a06ac: Deleting consensus metadata
I20260925 21:59:55.178936 20914 catalog_manager.cc:4841] TS bed8232116244564a8728ccf7f4a06ac (127.2.70.195:42275): tablet f9a137c1e7004887b2ff209a765e208b (table test-table [id=7f8ec1d5c3a244d49a5821c3945d4e00]) successfully deleted
I20260925 21:59:55.179965 21415 ts_tablet_manager.cc:1918] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:55.180034 21415 log.cc:1196] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/f9a137c1e7004887b2ff209a765e208b
I20260925 21:59:55.180300 21415 ts_tablet_manager.cc:1939] T f9a137c1e7004887b2ff209a765e208b P 56923b764ffc4cbdbcc391b4186bcf2a: Deleting consensus metadata
I20260925 21:59:55.180671 20914 catalog_manager.cc:4841] TS 56923b764ffc4cbdbcc391b4186bcf2a (127.2.70.194:41711): tablet f9a137c1e7004887b2ff209a765e208b (table test-table [id=7f8ec1d5c3a244d49a5821c3945d4e00]) successfully deleted
W20260925 21:59:55.181143 20912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:40299 (error 108)
I20260925 21:59:55.181401 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 21119
I20260925 21:59:55.182091 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:55.181476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:40299 (local address 127.0.0.1:39228)
0925 21:59:55.182024 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:55.182064 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:40299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525}
W20260925 21:59:55.182215 20912 catalog_manager.cc:4584] TS 5940d3aabce24248a3a1daf4cd4a2e99 (127.2.70.193:40299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9a137c1e7004887b2ff209a765e208b: Network error: Client connection negotiation failed: client connection to 127.2.70.193:40299: connect: Connection refused (error 111)
I20260925 21:59:55.186267 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 21250
I20260925 21:59:55.190227 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 20897
2026-09-25T21:59:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (664 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:59:55 dist-test-slave-2g0f krb5kdc[21421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:59:55 dist-test-slave-2g0f krb5kdc[21421](info): set up 2 sockets
Sep 25 21:59:55 dist-test-slave-2g0f krb5kdc[21421](info): commencing operation
krb5kdc: starting...
W20260925 21:59:56.155490 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:59:56 dist-test-slave-2g0f krb5kdc[21421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:59:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:59:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 21:59:56.274580 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:39977
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:39977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:56.354857 21434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:56.355079 21434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:56.355101 21434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:56.356444 21434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:59:56.356491 21434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:56.356504 21434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:56.356516 21434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:59:56.356527 21434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:59:56.358053 21434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:39977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:39977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:56.358295 21434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:56.358513 21434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:56.363237 21440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:56.363317 21439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:56.363343 21442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:56.363407 21434 server_base.cc:1034] running on GCE node
I20260925 21:59:56.363744 21434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:56.364035 21434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:56.365178 21434 hybrid_clock.cc:648] HybridClock initialized: now 1790373596365165 us; error 28 us; skew 500 ppm
Sep 25 21:59:56 dist-test-slave-2g0f krb5kdc[21421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:56.371898 21434 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 21:59:56.372366 21434 webserver.cc:460] Webserver started at http://127.2.70.254:42685/ using document root <none> and password file <none>
I20260925 21:59:56.372589 21434 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:56.372635 21434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:56.372749 21434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:56.373853 21434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "ddfe46d7c21f465c88e2fae4d37c6c33"
format_stamp: "Formatted at 2026-09-25 21:59:56 on dist-test-slave-2g0f"
server_key: "10cc6c18545a94c370c4c0b44311c6da"
server_key_iv: "9c0473da60ea72dbd768beeca8a1dcfd"
server_key_version: "encryptionkey@0"
I20260925 21:59:56.374182 21434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "ddfe46d7c21f465c88e2fae4d37c6c33"
format_stamp: "Formatted at 2026-09-25 21:59:56 on dist-test-slave-2g0f"
server_key: "10cc6c18545a94c370c4c0b44311c6da"
server_key_iv: "9c0473da60ea72dbd768beeca8a1dcfd"
server_key_version: "encryptionkey@0"
I20260925 21:59:56.375526 21434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:59:56.376391 21449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:56.376554 21434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 21:59:56.376678 21434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "ddfe46d7c21f465c88e2fae4d37c6c33"
format_stamp: "Formatted at 2026-09-25 21:59:56 on dist-test-slave-2g0f"
server_key: "10cc6c18545a94c370c4c0b44311c6da"
server_key_iv: "9c0473da60ea72dbd768beeca8a1dcfd"
server_key_version: "encryptionkey@0"
I20260925 21:59:56.376744 21434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:56.384874 21434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:56.386919 21434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:56.387051 21434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:56.390844 21434 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:39977
I20260925 21:59:56.390887 21501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:39977 every 8 connection(s)
I20260925 21:59:56.391204 21434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:59:56.392058 21502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:56.394346 21502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33: Bootstrap starting.
I20260925 21:59:56.394918 21502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:56.395184 21502 log.cc:824] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:56.395862 21502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33: No bootstrap required, opened a new log
I20260925 21:59:56.397454 21502 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfe46d7c21f465c88e2fae4d37c6c33" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39977 } }
I20260925 21:59:56.397600 21502 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:56.397624 21502 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfe46d7c21f465c88e2fae4d37c6c33, State: Initialized, Role: FOLLOWER
I20260925 21:59:56.397768 21502 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfe46d7c21f465c88e2fae4d37c6c33" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39977 } }
I20260925 21:59:56.397833 21502 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:59:56.397856 21502 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:59:56.397903 21502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:56.398468 21502 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfe46d7c21f465c88e2fae4d37c6c33" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39977 } }
I20260925 21:59:56.398559 21502 leader_election.cc:302] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddfe46d7c21f465c88e2fae4d37c6c33; no voters:
I20260925 21:59:56.398725 21502 leader_election.cc:288] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:59:56.398792 21505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:56.398979 21502 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:59:56.398993 21505 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [term 1 LEADER]: Becoming Leader. State: Replica: ddfe46d7c21f465c88e2fae4d37c6c33, State: Running, Role: LEADER
I20260925 21:59:56.399134 21505 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfe46d7c21f465c88e2fae4d37c6c33" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39977 } }
I20260925 21:59:56.399456 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 21434
I20260925 21:59:56.399569 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:59:56.399613 21506 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddfe46d7c21f465c88e2fae4d37c6c33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfe46d7c21f465c88e2fae4d37c6c33" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39977 } } }
I20260925 21:59:56.399688 21506 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:56.399748 2331 external_mini_cluster.cc:1428] Setting key 3ae646327e70bee95aeeea9e693becf0
I20260925 21:59:56.399863 21507 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddfe46d7c21f465c88e2fae4d37c6c33. Latest consensus state: current_term: 1 leader_uuid: "ddfe46d7c21f465c88e2fae4d37c6c33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfe46d7c21f465c88e2fae4d37c6c33" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 39977 } } }
I20260925 21:59:56.400005 21507 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33 [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:56.400113 21514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:59:56.400583 21514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:59:56.402091 21514 catalog_manager.cc:1347] Generated new cluster ID: 7edbb6efada140fb97433451a6052aa6
I20260925 21:59:56.402143 21514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 21:59:56 dist-test-slave-2g0f krb5kdc[21421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:56.411544 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.400388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.2.70.254:39977)
0925 21:59:56.400563 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:56.400573 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:56.400607 (+ 34us) server_negotiation.cc:407] Connection header received
0925 21:59:56.400922 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:56.400932 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:56.400993 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:56.401128 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:56.401806 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.403226 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.404342 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.404975 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.405081 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.407873 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:56.407877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:56.407886 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:56.407900 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:56.409834 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:56.410216 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:56.410220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:56.410220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:56.410355 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:56.410530 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:56.410531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:56.410531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:56.410869 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:56.411078 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:56.411153 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 21:59:56.411305 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260925 21:59:56.415370 21514 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:59:56.415993 21514 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:59:56.426757 21514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33: Generated new TSK 0
I20260925 21:59:56.426973 21514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
Entry for principal kudu/127.2.70.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab.
I20260925 21:59:56.462659 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39977
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:56.575239 21528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:56.575452 21528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:56.575476 21528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:56.576723 21528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:56.576771 21528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:56.576797 21528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:59:56.578222 21528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.193.keytab
--principal=kudu/127.2.70.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:56.578416 21528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:56.578589 21528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:56.583359 21536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:56.583503 21533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:56.583425 21534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:56.583738 21528 server_base.cc:1034] running on GCE node
I20260925 21:59:56.584019 21528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:56.584285 21528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:56.585449 21528 hybrid_clock.cc:648] HybridClock initialized: now 1790373596585431 us; error 30 us; skew 500 ppm
Sep 25 21:59:56 dist-test-slave-2g0f krb5kdc[21421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:56.587932 21528 init.cc:377] Logged in from keytab as kudu/127.2.70.193@KRBTEST.COM (short username kudu)
I20260925 21:59:56.588428 21528 webserver.cc:460] Webserver started at http://127.2.70.193:44665/ using document root <none> and password file <none>
I20260925 21:59:56.588656 21528 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:56.588702 21528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:56.588829 21528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:56.589871 21528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "bce010c44f434e7a896704df9fc7eaab"
format_stamp: "Formatted at 2026-09-25 21:59:56 on dist-test-slave-2g0f"
server_key: "898c378c00b82164a7b02c8a0233f347"
server_key_iv: "abe13ffd7f0ed402e9b509a6d7b03c84"
server_key_version: "encryptionkey@0"
I20260925 21:59:56.590204 21528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "bce010c44f434e7a896704df9fc7eaab"
format_stamp: "Formatted at 2026-09-25 21:59:56 on dist-test-slave-2g0f"
server_key: "898c378c00b82164a7b02c8a0233f347"
server_key_iv: "abe13ffd7f0ed402e9b509a6d7b03c84"
server_key_version: "encryptionkey@0"
I20260925 21:59:56.591435 21528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 21:59:56.592103 21543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:56.592247 21528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:56.592314 21528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "bce010c44f434e7a896704df9fc7eaab"
format_stamp: "Formatted at 2026-09-25 21:59:56 on dist-test-slave-2g0f"
server_key: "898c378c00b82164a7b02c8a0233f347"
server_key_iv: "abe13ffd7f0ed402e9b509a6d7b03c84"
server_key_version: "encryptionkey@0"
I20260925 21:59:56.592379 21528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:56.612640 21528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:56.614459 21528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:56.614578 21528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:56.614813 21528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:56.615137 21528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:56.615181 21528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:56.615213 21528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:56.615235 21528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:56.620657 21528 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:38787
I20260925 21:59:56.620707 21656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:38787 every 8 connection(s)
I20260925 21:59:56.621007 21528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
Sep 25 21:59:56 dist-test-slave-2g0f krb5kdc[21421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.193@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:56.628201 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 21528
I20260925 21:59:56.628314 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:59:56.628521 2331 external_mini_cluster.cc:1428] Setting key a3a61da62a920b4e8d9a06a02819d96d
I20260925 21:59:56.630246 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.621686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:58179 (local address 127.2.70.254:39977)
0925 21:59:56.621741 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:56.621743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:56.621935 (+ 192us) server_negotiation.cc:407] Connection header received
0925 21:59:56.622457 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:56.622459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:56.622474 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:56.622535 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:56.623549 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.624133 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.625198 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.625419 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.625478 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.627583 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:56.627585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:56.627585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:56.627593 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:56.628996 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:56.629355 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:56.629356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:56.629356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:56.629427 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:56.629638 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:56.629639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:56.629639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:56.629749 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:56.629804 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:56.630064 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 21:59:56.630151 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 21:59:56.630429 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.621657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:39977 (local address 127.2.70.193:58179)
0925 21:59:56.621820 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:56.621858 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:56.622341 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:56.622661 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:56.622672 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:56.623266 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:56.623465 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:56.623471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.624220 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.624221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:56.625111 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:56.625114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.625507 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.625508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:56.625622 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.626154 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:56.626162 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:56.627494 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:56.629065 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:56.629072 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:56.629075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:56.629289 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:56.629483 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:56.629484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:56.629484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:56.629586 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:56.629803 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:56.629809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:56.629951 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 21:59:56.630038 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 21:59:56.630204 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260925 21:59:56.630899 21657 heartbeater.cc:347] Connected to a master server at 127.2.70.254:39977
I20260925 21:59:56.630987 21657 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:56.631160 21657 heartbeater.cc:511] Master 127.2.70.254:39977 requested a full tablet report, sending...
I20260925 21:59:56.631706 21466 ts_manager.cc:194] Registered new tserver with Master: bce010c44f434e7a896704df9fc7eaab (127.2.70.193:38787)
I20260925 21:59:56.632587 21466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.193@KRBTEST.COM'} at 127.2.70.193:58179
WARNING: no policy specified for kudu/127.2.70.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
Entry for principal kudu/127.2.70.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab.
I20260925 21:59:56.653047 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39977
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:56.735049 21662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:56.735265 21662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:56.735288 21662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:56.736444 21662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:56.736481 21662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:56.736507 21662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:59:56.737896 21662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.194.keytab
--principal=kudu/127.2.70.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:56.738098 21662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:56.738281 21662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:56.743187 21668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:56.743194 21670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:56.743391 21662 server_base.cc:1034] running on GCE node
W20260925 21:59:56.743201 21667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:56.743669 21662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:56.743952 21662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:56.745102 21662 hybrid_clock.cc:648] HybridClock initialized: now 1790373596745083 us; error 28 us; skew 500 ppm
Sep 25 21:59:56 dist-test-slave-2g0f krb5kdc[21421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:56.747566 21662 init.cc:377] Logged in from keytab as kudu/127.2.70.194@KRBTEST.COM (short username kudu)
I20260925 21:59:56.748080 21662 webserver.cc:460] Webserver started at http://127.2.70.194:34107/ using document root <none> and password file <none>
I20260925 21:59:56.748301 21662 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:56.748351 21662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:56.748489 21662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:56.749560 21662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "cde0bd496b5945bdb5c5cd298aa018cf"
format_stamp: "Formatted at 2026-09-25 21:59:56 on dist-test-slave-2g0f"
server_key: "0b66a96086c6a7ac6214d5a1a3c2003d"
server_key_iv: "8b67913c40b91a87f10e30fd9ff59ed0"
server_key_version: "encryptionkey@0"
I20260925 21:59:56.749888 21662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "cde0bd496b5945bdb5c5cd298aa018cf"
format_stamp: "Formatted at 2026-09-25 21:59:56 on dist-test-slave-2g0f"
server_key: "0b66a96086c6a7ac6214d5a1a3c2003d"
server_key_iv: "8b67913c40b91a87f10e30fd9ff59ed0"
server_key_version: "encryptionkey@0"
I20260925 21:59:56.751250 21662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 21:59:56.752031 21677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:56.752189 21662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:56.752274 21662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "cde0bd496b5945bdb5c5cd298aa018cf"
format_stamp: "Formatted at 2026-09-25 21:59:56 on dist-test-slave-2g0f"
server_key: "0b66a96086c6a7ac6214d5a1a3c2003d"
server_key_iv: "8b67913c40b91a87f10e30fd9ff59ed0"
server_key_version: "encryptionkey@0"
I20260925 21:59:56.752336 21662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:56.766258 21662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:56.768455 21662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:56.768597 21662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:56.768817 21662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:56.769130 21662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:56.769162 21662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:56.769191 21662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:56.769209 21662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:56.774698 21662 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:44719
I20260925 21:59:56.774748 21790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:44719 every 8 connection(s)
I20260925 21:59:56.775050 21662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:59:56.777874 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 21662
I20260925 21:59:56.778023 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:59:56.778223 2331 external_mini_cluster.cc:1428] Setting key 214c834aacec8d86483eff8b89e82a17
Sep 25 21:59:56 dist-test-slave-2g0f krb5kdc[21421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.194@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:56.784368 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.775712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:41393 (local address 127.2.70.254:39977)
0925 21:59:56.775758 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:56.775761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:56.776010 (+ 249us) server_negotiation.cc:407] Connection header received
0925 21:59:56.776512 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:56.776514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:56.776527 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:56.776583 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:56.777579 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.778133 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.779256 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.779457 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.779513 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.781717 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:56.781718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:56.781719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:56.781727 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:56.783050 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:56.783450 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:56.783451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:56.783451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:56.783506 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:56.783725 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:56.783726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:56.783726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:56.783872 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:56.783932 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:56.784186 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 21:59:56.784274 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 21:59:56.784546 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.775693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:39977 (local address 127.2.70.194:41393)
0925 21:59:56.775873 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:56.775919 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:56.776414 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:56.776668 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:56.776673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:56.777269 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:56.777476 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:56.777481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.778222 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.778223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:56.779180 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:56.779183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.779523 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.779524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:56.779668 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.780184 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:56.780194 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:56.781630 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:56.783149 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:56.783157 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:56.783159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:56.783384 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:56.783561 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:56.783562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:56.783562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:56.783671 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:56.783920 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:56.783927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:56.784071 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 21:59:56.784161 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 21:59:56.784324 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260925 21:59:56.784999 21791 heartbeater.cc:347] Connected to a master server at 127.2.70.254:39977
I20260925 21:59:56.785111 21791 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:56.785310 21791 heartbeater.cc:511] Master 127.2.70.254:39977 requested a full tablet report, sending...
I20260925 21:59:56.785727 21466 ts_manager.cc:194] Registered new tserver with Master: cde0bd496b5945bdb5c5cd298aa018cf (127.2.70.194:44719)
WARNING: no policy specified for kudu/127.2.70.195@KRBTEST.COM; defaulting to no policy
I20260925 21:59:56.786231 21466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.194@KRBTEST.COM'} at 127.2.70.194:41393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
Entry for principal kudu/127.2.70.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab.
I20260925 21:59:56.807420 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39977
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:56.887195 21796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:56.887398 21796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:56.887423 21796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:56.888566 21796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:56.888603 21796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:56.888628 21796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:59:56.890022 21796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.195.keytab
--principal=kudu/127.2.70.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:39977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:56.890195 21796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:56.890372 21796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:56.895103 21804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:56.895208 21796 server_base.cc:1034] running on GCE node
W20260925 21:59:56.895162 21801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:56.895119 21802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:56.895535 21796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:56.895752 21796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:56.896899 21796 hybrid_clock.cc:648] HybridClock initialized: now 1790373596896886 us; error 24 us; skew 500 ppm
Sep 25 21:59:56 dist-test-slave-2g0f krb5kdc[21421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:56.899406 21796 init.cc:377] Logged in from keytab as kudu/127.2.70.195@KRBTEST.COM (short username kudu)
I20260925 21:59:56.899930 21796 webserver.cc:460] Webserver started at http://127.2.70.195:44709/ using document root <none> and password file <none>
I20260925 21:59:56.900103 21796 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:56.900141 21796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:56.900259 21796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:56.901328 21796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "a0482441a4dc44a286765d185fe780e3"
format_stamp: "Formatted at 2026-09-25 21:59:56 on dist-test-slave-2g0f"
server_key: "c3cc54b98a572d7e87f9c7b7ed678956"
server_key_iv: "7e0af66779c51076623aba6a9e94a3e8"
server_key_version: "encryptionkey@0"
I20260925 21:59:56.901656 21796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "a0482441a4dc44a286765d185fe780e3"
format_stamp: "Formatted at 2026-09-25 21:59:56 on dist-test-slave-2g0f"
server_key: "c3cc54b98a572d7e87f9c7b7ed678956"
server_key_iv: "7e0af66779c51076623aba6a9e94a3e8"
server_key_version: "encryptionkey@0"
I20260925 21:59:56.902877 21796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 21:59:56.903635 21811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:56.903801 21796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 21:59:56.903980 21796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "a0482441a4dc44a286765d185fe780e3"
format_stamp: "Formatted at 2026-09-25 21:59:56 on dist-test-slave-2g0f"
server_key: "c3cc54b98a572d7e87f9c7b7ed678956"
server_key_iv: "7e0af66779c51076623aba6a9e94a3e8"
server_key_version: "encryptionkey@0"
I20260925 21:59:56.904047 21796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:56.916467 21796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:56.918210 21796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:56.918336 21796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:56.918565 21796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:56.918851 21796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:56.918882 21796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:56.918919 21796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:56.918942 21796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:56.924499 21796 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:46419
I20260925 21:59:56.924551 21924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:46419 every 8 connection(s)
I20260925 21:59:56.924860 21796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
Sep 25 21:59:56 dist-test-slave-2g0f krb5kdc[21421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.195@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 21:59:56.932260 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 21796
I20260925 21:59:56.932341 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:59:56.932497 2331 external_mini_cluster.cc:1428] Setting key e9e67e93a07d0754add3ed9dc74da37c
I20260925 21:59:56.934054 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.925539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:37255 (local address 127.2.70.254:39977)
0925 21:59:56.925588 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:56.925590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:56.925766 (+ 176us) server_negotiation.cc:407] Connection header received
0925 21:59:56.926275 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:56.926276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:56.926290 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:56.926346 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:56.927420 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.927948 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.928931 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.929135 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.929190 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.931270 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:56.931272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:56.931272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:56.931278 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:56.932638 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:56.933212 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:56.933212 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:56.933213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:56.933278 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:56.933561 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:56.933561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:56.933561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:56.933667 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:56.933716 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:56.933930 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 21:59:56.933991 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 21:59:56.934230 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.925513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:39977 (local address 127.2.70.195:37255)
0925 21:59:56.925650 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:56.925690 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:56.926166 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:56.926416 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:56.926420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:56.927107 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:56.927328 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:56.927334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.928034 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.928035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:56.928849 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:56.928852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.929205 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.929206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:56.929334 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.929814 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 21:59:56.929823 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:56.931182 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:56.932718 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:56.932724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:56.932725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:56.933002 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:56.933366 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 21:59:56.933367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 21:59:56.933368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 21:59:56.933488 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 21:59:56.933714 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:56.933717 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:56.933835 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 21:59:56.933913 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 21:59:56.934041 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260925 21:59:56.934679 21925 heartbeater.cc:347] Connected to a master server at 127.2.70.254:39977
I20260925 21:59:56.934772 21925 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:56.934937 21925 heartbeater.cc:511] Master 127.2.70.254:39977 requested a full tablet report, sending...
I20260925 21:59:56.935324 21466 ts_manager.cc:194] Registered new tserver with Master: a0482441a4dc44a286765d185fe780e3 (127.2.70.195:46419)
I20260925 21:59:56.935674 21466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.195@KRBTEST.COM'} at 127.2.70.195:37255
I20260925 21:59:56.944187 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:59:56.950923 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.945444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.2.70.254:39977)
0925 21:59:56.945512 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:56.945514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:56.945561 (+ 47us) server_negotiation.cc:407] Connection header received
0925 21:59:56.945622 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:56.945624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:56.945634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:56.945679 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:56.946238 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.946746 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.947437 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.947646 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.947698 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.948825 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:56.948827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:56.948827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:56.948834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:56.949834 (+ 1000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:56.950175 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:56.950176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:56.950176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:56.950231 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:56.950385 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:56.950386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:56.950386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:56.950540 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:56.950637 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:56.950756 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 21:59:56.950832 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 21:59:56.952430 21466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:59:56.953989 21466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:59:56.960057 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.956260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:38787 (local address 127.0.0.1:48114)
0925 21:59:56.956419 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:56.956458 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:56.956519 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:56.956787 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:56.956791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:56.956793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:56.956880 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:56.956883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.957946 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.957947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:56.958876 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:56.958878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.959693 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.959694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:56.959776 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.959791 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 21:59:56.959914 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 21:59:56.959973 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260925 21:59:56.960059 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.956371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.2.70.193:38787)
0925 21:59:56.956610 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:56.956615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:56.956627 (+ 12us) server_negotiation.cc:407] Connection header received
0925 21:59:56.956670 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:56.956672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:56.956689 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:56.956781 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:56.956985 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.957831 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.958961 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.959615 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.959684 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.959741 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:56.959930 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 21:59:56.959976 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260925 21:59:56.961074 21591 tablet_service.cc:1467] Processing CreateTablet for tablet 23add715e14543ecaa973e5eb5a43561 (DEFAULT_TABLE table=test-table [id=a60eacb3201042cf9cb51e48d2b72762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:56.961472 21591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23add715e14543ecaa973e5eb5a43561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:56.961884 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.956582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.2.70.195:46419)
0925 21:59:56.956774 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:56.956779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:56.957007 (+ 228us) server_negotiation.cc:407] Connection header received
0925 21:59:56.957055 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:56.957058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:56.957074 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:56.957144 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:56.957359 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.958484 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.960791 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.961477 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.961544 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.961599 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:56.961782 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 21:59:56.961820 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20260925 21:59:56.963992 21942 tablet_bootstrap.cc:492] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab: Bootstrap starting.
I20260925 21:59:56.964167 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.956649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.2.70.194:44719)
0925 21:59:56.956815 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:56.956819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:56.956929 (+ 110us) server_negotiation.cc:407] Connection header received
0925 21:59:56.957065 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:56.957068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:56.957084 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:56.957149 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:56.957366 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.958490 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.959603 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.960308 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.960372 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.960427 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:56.964048 (+ 3621us) server_negotiation.cc:299] Negotiation successful
0925 21:59:56.964098 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":70,"threads_started":1}
I20260925 21:59:56.960614 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.956262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:44719 (local address 127.0.0.1:53616)
0925 21:59:56.956852 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:56.956862 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:56.956970 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:56.957235 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:56.957236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:56.957238 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:56.957287 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:56.957288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.958594 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.958595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:56.959504 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:56.959506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.960390 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.960391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:56.960455 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.960459 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:59:56.960505 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 21:59:56.960548 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":106,"threads_started":1}
I20260925 21:59:56.964468 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:56.956496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:46419 (local address 127.0.0.1:50038)
0925 21:59:56.956886 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:56.956896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:56.956945 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:56.957227 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:56.957228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:56.957230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:56.957272 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:56.957274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.959595 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.959595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:56.960706 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:56.960709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:56.961634 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:56.961635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:56.961695 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:56.961699 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:59:56.961747 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 21:59:56.961782 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":78,"threads_started":1}
I20260925 21:59:56.964825 21942 tablet_bootstrap.cc:654] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:56.964974 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 23add715e14543ecaa973e5eb5a43561 (DEFAULT_TABLE table=test-table [id=a60eacb3201042cf9cb51e48d2b72762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:56.965171 21859 tablet_service.cc:1467] Processing CreateTablet for tablet 23add715e14543ecaa973e5eb5a43561 (DEFAULT_TABLE table=test-table [id=a60eacb3201042cf9cb51e48d2b72762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:56.965298 21725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23add715e14543ecaa973e5eb5a43561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:56.965299 21942 log.cc:824] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:56.965456 21859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23add715e14543ecaa973e5eb5a43561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:56.965978 21942 tablet_bootstrap.cc:492] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab: No bootstrap required, opened a new log
I20260925 21:59:56.966038 21942 ts_tablet_manager.cc:1397] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:59:56.967373 21942 raft_consensus.cc:348] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde0bd496b5945bdb5c5cd298aa018cf" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44719 } } peers { permanent_uuid: "bce010c44f434e7a896704df9fc7eaab" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38787 } } peers { permanent_uuid: "a0482441a4dc44a286765d185fe780e3" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46419 } }
I20260925 21:59:56.967517 21942 raft_consensus.cc:374] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:56.967549 21942 raft_consensus.cc:729] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce010c44f434e7a896704df9fc7eaab, State: Initialized, Role: FOLLOWER
I20260925 21:59:56.967695 21942 consensus_queue.cc:260] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde0bd496b5945bdb5c5cd298aa018cf" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44719 } } peers { permanent_uuid: "bce010c44f434e7a896704df9fc7eaab" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38787 } } peers { permanent_uuid: "a0482441a4dc44a286765d185fe780e3" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46419 } }
I20260925 21:59:56.967991 21657 heartbeater.cc:503] Master 127.2.70.254:39977 was elected leader, sending a full tablet report...
I20260925 21:59:56.968240 21942 ts_tablet_manager.cc:1428] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 21:59:56.968456 21945 tablet_bootstrap.cc:492] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf: Bootstrap starting.
I20260925 21:59:56.968950 21946 tablet_bootstrap.cc:492] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3: Bootstrap starting.
I20260925 21:59:56.969079 21945 tablet_bootstrap.cc:654] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:56.969581 21946 tablet_bootstrap.cc:654] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:56.969620 21945 log.cc:824] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:56.970069 21946 log.cc:824] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:56.970397 21945 tablet_bootstrap.cc:492] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf: No bootstrap required, opened a new log
I20260925 21:59:56.970449 21945 ts_tablet_manager.cc:1397] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:59:56.970772 21946 tablet_bootstrap.cc:492] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3: No bootstrap required, opened a new log
I20260925 21:59:56.970836 21946 ts_tablet_manager.cc:1397] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:59:56.972321 21945 raft_consensus.cc:348] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde0bd496b5945bdb5c5cd298aa018cf" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44719 } } peers { permanent_uuid: "bce010c44f434e7a896704df9fc7eaab" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38787 } } peers { permanent_uuid: "a0482441a4dc44a286765d185fe780e3" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46419 } }
I20260925 21:59:56.972499 21945 raft_consensus.cc:374] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:56.972543 21945 raft_consensus.cc:729] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde0bd496b5945bdb5c5cd298aa018cf, State: Initialized, Role: FOLLOWER
I20260925 21:59:56.972656 21945 consensus_queue.cc:260] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde0bd496b5945bdb5c5cd298aa018cf" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44719 } } peers { permanent_uuid: "bce010c44f434e7a896704df9fc7eaab" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38787 } } peers { permanent_uuid: "a0482441a4dc44a286765d185fe780e3" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46419 } }
I20260925 21:59:56.972632 21946 raft_consensus.cc:348] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde0bd496b5945bdb5c5cd298aa018cf" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44719 } } peers { permanent_uuid: "bce010c44f434e7a896704df9fc7eaab" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38787 } } peers { permanent_uuid: "a0482441a4dc44a286765d185fe780e3" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46419 } }
I20260925 21:59:56.972788 21946 raft_consensus.cc:374] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:56.972833 21946 raft_consensus.cc:729] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0482441a4dc44a286765d185fe780e3, State: Initialized, Role: FOLLOWER
I20260925 21:59:56.972921 21945 ts_tablet_manager.cc:1428] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 21:59:56.972996 21946 consensus_queue.cc:260] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde0bd496b5945bdb5c5cd298aa018cf" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44719 } } peers { permanent_uuid: "bce010c44f434e7a896704df9fc7eaab" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38787 } } peers { permanent_uuid: "a0482441a4dc44a286765d185fe780e3" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46419 } }
I20260925 21:59:56.972924 21791 heartbeater.cc:503] Master 127.2.70.254:39977 was elected leader, sending a full tablet report...
I20260925 21:59:56.973304 21925 heartbeater.cc:503] Master 127.2.70.254:39977 was elected leader, sending a full tablet report...
I20260925 21:59:56.973361 21946 ts_tablet_manager.cc:1428] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260925 21:59:57.025671 21792 tablet.cc:2369] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:59:57.092739 21950 raft_consensus.cc:482] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:59:57.092861 21950 raft_consensus.cc:504] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde0bd496b5945bdb5c5cd298aa018cf" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44719 } } peers { permanent_uuid: "bce010c44f434e7a896704df9fc7eaab" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38787 } } peers { permanent_uuid: "a0482441a4dc44a286765d185fe780e3" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46419 } }
I20260925 21:59:57.093194 21950 leader_election.cc:288] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cde0bd496b5945bdb5c5cd298aa018cf (127.2.70.194:44719), bce010c44f434e7a896704df9fc7eaab (127.2.70.193:38787)
I20260925 21:59:57.097147 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.093566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:48745 (local address 127.2.70.194:44719)
0925 21:59:57.093613 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.093615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.093624 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:59:57.093652 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.093653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.093665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.093725 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:57.093880 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.094668 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.096192 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.096763 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.096838 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.096847 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.097033 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.097091 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 21:59:57.097131 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.093474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:60889 (local address 127.2.70.193:38787)
0925 21:59:57.093527 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.093529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.093656 (+ 127us) server_negotiation.cc:407] Connection header received
0925 21:59:57.093713 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.093714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.093725 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.093775 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:57.093890 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.094643 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.096191 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.096763 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.096838 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.096847 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.097018 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.097059 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 21:59:57.097160 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.093395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:44719 (local address 127.2.70.195:48745)
0925 21:59:57.093448 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:57.093461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:57.093537 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:57.093736 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:57.093737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:57.093739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:57.093811 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.093814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.094725 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.094726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.096104 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.096108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.096840 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.096841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.096944 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.096950 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 21:59:57.097004 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 21:59:57.097053 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":71}
I20260925 21:59:57.097165 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.093395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:38787 (local address 127.2.70.195:60889)
0925 21:59:57.093604 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:57.093612 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:57.093660 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:57.093781 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:57.093782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:57.093783 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:57.093835 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.093836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.094725 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.094726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.096120 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.096121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.096840 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.096841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.096944 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.096950 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 21:59:57.097004 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 21:59:57.097053 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":125,"thread_start_us":81,"threads_started":1}
I20260925 21:59:57.097523 21745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23add715e14543ecaa973e5eb5a43561" candidate_uuid: "a0482441a4dc44a286765d185fe780e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde0bd496b5945bdb5c5cd298aa018cf" is_pre_election: true
I20260925 21:59:57.097545 21611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23add715e14543ecaa973e5eb5a43561" candidate_uuid: "a0482441a4dc44a286765d185fe780e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce010c44f434e7a896704df9fc7eaab" is_pre_election: true
I20260925 21:59:57.097685 21745 raft_consensus.cc:2454] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0482441a4dc44a286765d185fe780e3 in term 0.
I20260925 21:59:57.097687 21611 raft_consensus.cc:2454] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0482441a4dc44a286765d185fe780e3 in term 0.
I20260925 21:59:57.097896 21813 leader_election.cc:302] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0482441a4dc44a286765d185fe780e3, bce010c44f434e7a896704df9fc7eaab; no voters:
I20260925 21:59:57.098058 21950 raft_consensus.cc:2781] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:59:57.098115 21950 raft_consensus.cc:482] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:59:57.098143 21950 raft_consensus.cc:3037] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:57.098834 21950 raft_consensus.cc:504] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde0bd496b5945bdb5c5cd298aa018cf" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44719 } } peers { permanent_uuid: "bce010c44f434e7a896704df9fc7eaab" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38787 } } peers { permanent_uuid: "a0482441a4dc44a286765d185fe780e3" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46419 } }
I20260925 21:59:57.098984 21950 leader_election.cc:288] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [CANDIDATE]: Term 1 election: Requested vote from peers cde0bd496b5945bdb5c5cd298aa018cf (127.2.70.194:44719), bce010c44f434e7a896704df9fc7eaab (127.2.70.193:38787)
I20260925 21:59:57.099154 21745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23add715e14543ecaa973e5eb5a43561" candidate_uuid: "a0482441a4dc44a286765d185fe780e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde0bd496b5945bdb5c5cd298aa018cf"
I20260925 21:59:57.099153 21611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23add715e14543ecaa973e5eb5a43561" candidate_uuid: "a0482441a4dc44a286765d185fe780e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce010c44f434e7a896704df9fc7eaab"
I20260925 21:59:57.099222 21745 raft_consensus.cc:3037] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:57.099238 21611 raft_consensus.cc:3037] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:57.100011 21745 raft_consensus.cc:2454] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0482441a4dc44a286765d185fe780e3 in term 1.
I20260925 21:59:57.100119 21611 raft_consensus.cc:2454] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0482441a4dc44a286765d185fe780e3 in term 1.
I20260925 21:59:57.100188 21812 leader_election.cc:302] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0482441a4dc44a286765d185fe780e3, cde0bd496b5945bdb5c5cd298aa018cf; no voters:
I20260925 21:59:57.100270 21950 raft_consensus.cc:2781] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:57.100440 21950 raft_consensus.cc:686] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 1 LEADER]: Becoming Leader. State: Replica: a0482441a4dc44a286765d185fe780e3, State: Running, Role: LEADER
I20260925 21:59:57.100528 21950 consensus_queue.cc:237] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde0bd496b5945bdb5c5cd298aa018cf" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44719 } } peers { permanent_uuid: "bce010c44f434e7a896704df9fc7eaab" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38787 } } peers { permanent_uuid: "a0482441a4dc44a286765d185fe780e3" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46419 } }
I20260925 21:59:57.101344 21466 catalog_manager.cc:5495] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0482441a4dc44a286765d185fe780e3 (127.2.70.195). New cstate: current_term: 1 leader_uuid: "a0482441a4dc44a286765d185fe780e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde0bd496b5945bdb5c5cd298aa018cf" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bce010c44f434e7a896704df9fc7eaab" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0482441a4dc44a286765d185fe780e3" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 46419 } health_report { overall_health: HEALTHY } } }
W20260925 21:59:57.121758 21658 tablet.cc:2369] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 21:59:57.143189 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.140133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.2.70.195:46419)
0925 21:59:57.140177 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.140179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.140188 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:59:57.140259 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.140261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.140284 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.140361 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:59:57.140565 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.141281 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.142325 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.142626 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.142698 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.142817 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:59:57.142962 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:59:57.143038 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.143085 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.143105 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 21:59:57.144999 21745 raft_consensus.cc:1261] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf [term 1 FOLLOWER]: Refusing update from remote peer a0482441a4dc44a286765d185fe780e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:57.145052 21611 raft_consensus.cc:1261] T 23add715e14543ecaa973e5eb5a43561 P bce010c44f434e7a896704df9fc7eaab [term 1 FOLLOWER]: Refusing update from remote peer a0482441a4dc44a286765d185fe780e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:57.145334 21952 consensus_queue.cc:1035] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce010c44f434e7a896704df9fc7eaab" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 38787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:57.145467 21950 consensus_queue.cc:1035] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde0bd496b5945bdb5c5cd298aa018cf" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 44719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:57.150244 21466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:57.150339 21466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:57.151027 21466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddfe46d7c21f465c88e2fae4d37c6c33: Sending DeleteTablet for 3 replicas of tablet 23add715e14543ecaa973e5eb5a43561
I20260925 21:59:57.151332 21725 tablet_service.cc:1514] Processing DeleteTablet for tablet 23add715e14543ecaa973e5eb5a43561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:57 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:53616
I20260925 21:59:57.151437 21859 tablet_service.cc:1514] Processing DeleteTablet for tablet 23add715e14543ecaa973e5eb5a43561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:57 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:50038
I20260925 21:59:57.151453 21591 tablet_service.cc:1514] Processing DeleteTablet for tablet 23add715e14543ecaa973e5eb5a43561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:57 UTC) from {username='kudu', principal='kudu/127.2.70.254@KRBTEST.COM'} at 127.0.0.1:48114
I20260925 21:59:57.151549 21966 tablet_replica.cc:330] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf: stopping tablet replica
I20260925 21:59:57.151638 21966 raft_consensus.cc:2229] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:59:57.151656 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 21528
I20260925 21:59:57.151732 21966 raft_consensus.cc:2258] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:57.152444 21966 ts_tablet_manager.cc:1905] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:57.154060 21966 ts_tablet_manager.cc:1918] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:57.154129 21966 log.cc:1196] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/23add715e14543ecaa973e5eb5a43561
I20260925 21:59:57.154389 21966 ts_tablet_manager.cc:1939] T 23add715e14543ecaa973e5eb5a43561 P cde0bd496b5945bdb5c5cd298aa018cf: Deleting consensus metadata
I20260925 21:59:57.154790 21450 catalog_manager.cc:4841] TS cde0bd496b5945bdb5c5cd298aa018cf (127.2.70.194:44719): tablet 23add715e14543ecaa973e5eb5a43561 (table test-table [id=a60eacb3201042cf9cb51e48d2b72762]) successfully deleted
W20260925 21:59:57.158515 21451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:38787 (error 108)
I20260925 21:59:57.158628 21968 tablet_replica.cc:330] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3: stopping tablet replica
I20260925 21:59:57.158740 21968 raft_consensus.cc:2229] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 21:59:57.158849 21968 raft_consensus.cc:2258] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:57.159037 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.158904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:38787 (local address 127.0.0.1:48124)
0925 21:59:57.158949 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:57.159003 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:38787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 21:59:57.159148 21451 catalog_manager.cc:4584] TS bce010c44f434e7a896704df9fc7eaab (127.2.70.193:38787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 23add715e14543ecaa973e5eb5a43561: Network error: Client connection negotiation failed: client connection to 127.2.70.193:38787: connect: Connection refused (error 111)
I20260925 21:59:57.159232 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 21662
I20260925 21:59:57.159747 21968 ts_tablet_manager.cc:1905] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:57.161362 21968 ts_tablet_manager.cc:1918] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:57.161433 21968 log.cc:1196] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/23add715e14543ecaa973e5eb5a43561
I20260925 21:59:57.161725 21968 ts_tablet_manager.cc:1939] T 23add715e14543ecaa973e5eb5a43561 P a0482441a4dc44a286765d185fe780e3: Deleting consensus metadata
I20260925 21:59:57.162261 21451 catalog_manager.cc:4841] TS a0482441a4dc44a286765d185fe780e3 (127.2.70.195:46419): tablet 23add715e14543ecaa973e5eb5a43561 (table test-table [id=a60eacb3201042cf9cb51e48d2b72762]) successfully deleted
I20260925 21:59:57.164623 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 21796
I20260925 21:59:57.168453 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 21434
2026-09-25T21:59:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1979 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T21:59:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:59:57Z Disabled control of system clock
I20260925 21:59:57.193926 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:34617
--webserver_interface=127.2.70.254
--webserver_port=0
--builtin_ntp_servers=127.2.70.212:39279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:34617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 21:59:57.273109 21973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:57.273329 21973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:57.273347 21973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:57.274524 21973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:59:57.274566 21973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:57.274578 21973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:57.274590 21973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:59:57.274600 21973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:59:57.275965 21973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:39279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:34617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:34617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:57.276194 21973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:57.276476 21973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:57.281239 21978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:57.281286 21973 server_base.cc:1034] running on GCE node
W20260925 21:59:57.281265 21979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:57.281248 21981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:57.281605 21973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:57.281884 21973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:57.283051 21973 hybrid_clock.cc:648] HybridClock initialized: now 1790373597283037 us; error 29 us; skew 500 ppm
I20260925 21:59:57.284360 21973 webserver.cc:460] Webserver started at http://127.2.70.254:32941/ using document root <none> and password file <none>
I20260925 21:59:57.284576 21973 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:57.284622 21973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:57.284749 21973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:57.285791 21973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "18538e883e894db581e2b28279b8ee1d"
format_stamp: "Formatted at 2026-09-25 21:59:57 on dist-test-slave-2g0f"
server_key: "828cb6945e2746dfbd85011c32bd613a"
server_key_iv: "605bdb87c32d37414fb43f5d73d68ce8"
server_key_version: "encryptionkey@0"
I20260925 21:59:57.286126 21973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "18538e883e894db581e2b28279b8ee1d"
format_stamp: "Formatted at 2026-09-25 21:59:57 on dist-test-slave-2g0f"
server_key: "828cb6945e2746dfbd85011c32bd613a"
server_key_iv: "605bdb87c32d37414fb43f5d73d68ce8"
server_key_version: "encryptionkey@0"
I20260925 21:59:57.287370 21973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 21:59:57.288064 21987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:57.288245 21973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:57.288331 21973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "18538e883e894db581e2b28279b8ee1d"
format_stamp: "Formatted at 2026-09-25 21:59:57 on dist-test-slave-2g0f"
server_key: "828cb6945e2746dfbd85011c32bd613a"
server_key_iv: "605bdb87c32d37414fb43f5d73d68ce8"
server_key_version: "encryptionkey@0"
I20260925 21:59:57.288393 21973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:57.305943 21973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:57.306277 21973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:57.306401 21973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:57.310129 21973 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:34617
I20260925 21:59:57.310160 22039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:34617 every 8 connection(s)
I20260925 21:59:57.310474 21973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:59:57.311362 22040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:57.313591 22040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d: Bootstrap starting.
I20260925 21:59:57.314110 22040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:57.314359 22040 log.cc:824] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:57.314939 22040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d: No bootstrap required, opened a new log
I20260925 21:59:57.316555 22040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18538e883e894db581e2b28279b8ee1d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34617 } }
I20260925 21:59:57.316709 22040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:57.316732 22040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18538e883e894db581e2b28279b8ee1d, State: Initialized, Role: FOLLOWER
I20260925 21:59:57.316869 22040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18538e883e894db581e2b28279b8ee1d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34617 } }
I20260925 21:59:57.316926 22040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:59:57.316947 22040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:59:57.316991 22040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:57.317464 22040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18538e883e894db581e2b28279b8ee1d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34617 } }
I20260925 21:59:57.317549 22040 leader_election.cc:302] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18538e883e894db581e2b28279b8ee1d; no voters:
I20260925 21:59:57.317703 22040 leader_election.cc:288] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:59:57.317775 22043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:57.317893 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 21973
I20260925 21:59:57.317943 22040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 21:59:57.317992 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:59:57.318039 22043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [term 1 LEADER]: Becoming Leader. State: Replica: 18538e883e894db581e2b28279b8ee1d, State: Running, Role: LEADER
I20260925 21:59:57.318140 22043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18538e883e894db581e2b28279b8ee1d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34617 } }
I20260925 21:59:57.318183 2331 external_mini_cluster.cc:1428] Setting key a8a69cbe740d6cf597af2b3618974b10
I20260925 21:59:57.318729 22045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18538e883e894db581e2b28279b8ee1d. Latest consensus state: current_term: 1 leader_uuid: "18538e883e894db581e2b28279b8ee1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18538e883e894db581e2b28279b8ee1d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34617 } } }
I20260925 21:59:57.318810 22045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:57.319027 22044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18538e883e894db581e2b28279b8ee1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18538e883e894db581e2b28279b8ee1d" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34617 } } }
I20260925 21:59:57.319109 22054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:59:57.319135 22044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:57.319571 22054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:59:57.320979 22054 catalog_manager.cc:1347] Generated new cluster ID: 2f963e2616f0482abf5757e71cfb62ba
I20260925 21:59:57.321030 22054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:59:57.324687 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.318659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.2.70.254:34617)
0925 21:59:57.318934 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.318943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.319455 (+ 512us) server_negotiation.cc:407] Connection header received
0925 21:59:57.319606 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.319613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.319686 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.319820 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:57.320155 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.321478 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.322544 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.323170 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.323287 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.324005 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:57.324009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:57.324009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:57.324215 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:57.324256 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:57.324272 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:57.324340 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.324394 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.324470 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
I20260925 21:59:57.346709 22054 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:59:57.347446 22054 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:59:57.354637 22054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d: Generated new TSK 0
I20260925 21:59:57.354828 22054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:59:57.360287 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.193:0
--local_ip_for_outbound_sockets=127.2.70.193
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34617
--builtin_ntp_servers=127.2.70.212:39279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 21:59:57.439966 22064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:57.440164 22064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:57.440193 22064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:57.441336 22064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:57.441373 22064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:57.441406 22064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.193
I20260925 21:59:57.442729 22064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:39279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.193
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.193
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:57.442960 22064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:57.443229 22064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:57.448036 22070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:57.448029 22072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:57.448029 22069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:57.448104 22064 server_base.cc:1034] running on GCE node
I20260925 21:59:57.448397 22064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:57.448609 22064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:57.449744 22064 hybrid_clock.cc:648] HybridClock initialized: now 1790373597449729 us; error 29 us; skew 500 ppm
I20260925 21:59:57.451053 22064 webserver.cc:460] Webserver started at http://127.2.70.193:41885/ using document root <none> and password file <none>
I20260925 21:59:57.451277 22064 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:57.451326 22064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:57.451457 22064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:57.452572 22064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/data/instance:
uuid: "eb9b6fd76ff648febfc13a6ef421966e"
format_stamp: "Formatted at 2026-09-25 21:59:57 on dist-test-slave-2g0f"
server_key: "ea238b40cdce0169aaa8e01a3f3ebf38"
server_key_iv: "4330ccedb86c6cc5ff7e1e8fd4924a2a"
server_key_version: "encryptionkey@0"
I20260925 21:59:57.452898 22064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance:
uuid: "eb9b6fd76ff648febfc13a6ef421966e"
format_stamp: "Formatted at 2026-09-25 21:59:57 on dist-test-slave-2g0f"
server_key: "ea238b40cdce0169aaa8e01a3f3ebf38"
server_key_iv: "4330ccedb86c6cc5ff7e1e8fd4924a2a"
server_key_version: "encryptionkey@0"
I20260925 21:59:57.454101 22064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 21:59:57.454779 22078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:57.455001 22064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 21:59:57.455082 22064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/wal
uuid: "eb9b6fd76ff648febfc13a6ef421966e"
format_stamp: "Formatted at 2026-09-25 21:59:57 on dist-test-slave-2g0f"
server_key: "ea238b40cdce0169aaa8e01a3f3ebf38"
server_key_iv: "4330ccedb86c6cc5ff7e1e8fd4924a2a"
server_key_version: "encryptionkey@0"
I20260925 21:59:57.455149 22064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:57.463662 22064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:57.463985 22064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:57.464109 22064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:57.464309 22064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:57.464609 22064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:57.464646 22064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:57.464675 22064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:57.464699 22064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:57.469892 22064 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.193:46697
I20260925 21:59:57.469965 22191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.193:46697 every 8 connection(s)
I20260925 21:59:57.470255 22064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/data/info.pb
I20260925 21:59:57.474689 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 22064
I20260925 21:59:57.474787 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-0/wal/instance
I20260925 21:59:57.474988 2331 external_mini_cluster.cc:1428] Setting key c009a16ae7e42b438082ca3015149512
I20260925 21:59:57.475383 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.470970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:50039 (local address 127.2.70.254:34617)
0925 21:59:57.471047 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.471049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.471288 (+ 239us) server_negotiation.cc:407] Connection header received
0925 21:59:57.471785 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.471787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.471800 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.471869 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:57.472526 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.473024 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.474021 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.474210 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.474264 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.475107 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:57.475108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:57.475108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:57.475113 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:57.475130 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:57.475133 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:57.475182 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.475275 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.475308 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 21:59:57.475574 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.470982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34617 (local address 127.2.70.193:50039)
0925 21:59:57.471175 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:57.471209 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:57.471696 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:57.471948 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:57.471952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:57.471960 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:57.472447 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.472452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.473108 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.473109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.473939 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.473942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.474277 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.474277 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.474385 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.474964 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:59:57.474964 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:57.474998 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:59:57.475022 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:57.475185 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:57.475190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:57.475193 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:57.475257 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 21:59:57.475359 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260925 21:59:57.476013 22192 heartbeater.cc:347] Connected to a master server at 127.2.70.254:34617
I20260925 21:59:57.476109 22192 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:57.476136 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.194:0
--local_ip_for_outbound_sockets=127.2.70.194
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34617
--builtin_ntp_servers=127.2.70.212:39279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 21:59:57.476301 22192 heartbeater.cc:511] Master 127.2.70.254:34617 requested a full tablet report, sending...
I20260925 21:59:57.476781 22004 ts_manager.cc:194] Registered new tserver with Master: eb9b6fd76ff648febfc13a6ef421966e (127.2.70.193:46697)
I20260925 21:59:57.477377 22004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.193:50039
W20260925 21:59:57.554603 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:57.554798 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:57.554818 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:57.555999 22195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:57.556041 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:57.556066 22195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.194
I20260925 21:59:57.557564 22195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:39279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.194
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.194
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:57.557811 22195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:57.558087 22195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:57.562986 22203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:57.562994 22201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:57.563135 22195 server_base.cc:1034] running on GCE node
W20260925 21:59:57.563225 22200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:57.563441 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:57.563658 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:57.564800 22195 hybrid_clock.cc:648] HybridClock initialized: now 1790373597564768 us; error 44 us; skew 500 ppm
I20260925 21:59:57.566054 22195 webserver.cc:460] Webserver started at http://127.2.70.194:43241/ using document root <none> and password file <none>
I20260925 21:59:57.566278 22195 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:57.566323 22195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:57.566447 22195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:57.567489 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/data/instance:
uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8"
format_stamp: "Formatted at 2026-09-25 21:59:57 on dist-test-slave-2g0f"
server_key: "dacd046f2e8e90b998ba12a41a945995"
server_key_iv: "c9a4b6b1fdc6e1e713e23fbab40dccaa"
server_key_version: "encryptionkey@0"
I20260925 21:59:57.567809 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance:
uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8"
format_stamp: "Formatted at 2026-09-25 21:59:57 on dist-test-slave-2g0f"
server_key: "dacd046f2e8e90b998ba12a41a945995"
server_key_iv: "c9a4b6b1fdc6e1e713e23fbab40dccaa"
server_key_version: "encryptionkey@0"
I20260925 21:59:57.569017 22195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 21:59:57.569657 22209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:57.569794 22195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:57.569864 22195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/wal
uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8"
format_stamp: "Formatted at 2026-09-25 21:59:57 on dist-test-slave-2g0f"
server_key: "dacd046f2e8e90b998ba12a41a945995"
server_key_iv: "c9a4b6b1fdc6e1e713e23fbab40dccaa"
server_key_version: "encryptionkey@0"
I20260925 21:59:57.569931 22195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:57.582971 22195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:57.583273 22195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:57.583398 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:57.583614 22195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:57.583989 22195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:57.584028 22195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:57.584059 22195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:57.584079 22195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:57.589563 22195 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.194:37365
I20260925 21:59:57.589615 22322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.194:37365 every 8 connection(s)
I20260925 21:59:57.589920 22195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/data/info.pb
I20260925 21:59:57.590803 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 22195
I20260925 21:59:57.590904 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/wal/instance
I20260925 21:59:57.591094 2331 external_mini_cluster.cc:1428] Setting key f0e72e4504a4ba93b290388e30be73bf
I20260925 21:59:57.592015 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.195:0
--local_ip_for_outbound_sockets=127.2.70.195
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34617
--builtin_ntp_servers=127.2.70.212:39279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 21:59:57.595263 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.590640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.194:48387 (local address 127.2.70.254:34617)
0925 21:59:57.590693 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.590695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.590927 (+ 232us) server_negotiation.cc:407] Connection header received
0925 21:59:57.591560 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.591561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.591580 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.591635 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:57.592418 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.592905 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.593862 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.594131 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.594192 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.594970 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:57.594971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:57.594971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:57.594976 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:57.595003 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:57.595008 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:57.595060 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.595149 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.595179 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 21:59:57.595415 22325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.590637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34617 (local address 127.2.70.194:48387)
0925 21:59:57.590815 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:57.590849 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:57.591473 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:57.591705 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:57.591719 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:57.591730 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:57.592339 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.592345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.592983 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.592984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.593779 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.593781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.594203 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.594204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.594327 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.594836 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:59:57.594837 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:57.594866 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:59:57.594889 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:57.595071 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:57.595077 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:57.595080 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:57.595143 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 21:59:57.595229 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260925 21:59:57.595860 22323 heartbeater.cc:347] Connected to a master server at 127.2.70.254:34617
I20260925 21:59:57.595958 22323 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:57.596144 22323 heartbeater.cc:511] Master 127.2.70.254:34617 requested a full tablet report, sending...
I20260925 21:59:57.596482 22004 ts_manager.cc:194] Registered new tserver with Master: 6433d546b0cb4c5b9d9d71bcb3a3d8d8 (127.2.70.194:37365)
I20260925 21:59:57.596841 22004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.194:48387
W20260925 21:59:57.671358 22326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:57.671568 22326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:57.671598 22326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:57.672817 22326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:57.672853 22326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:57.672887 22326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.195
I20260925 21:59:57.674238 22326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:39279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.195
--webserver_port=0
--tserver_master_addrs=127.2.70.254:34617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.195
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:57.674459 22326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:57.674726 22326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:57.679633 22331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:57.679633 22334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:57.679672 22326 server_base.cc:1034] running on GCE node
W20260925 21:59:57.679664 22332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:57.680080 22326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:57.680325 22326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:57.681485 22326 hybrid_clock.cc:648] HybridClock initialized: now 1790373597681469 us; error 27 us; skew 500 ppm
I20260925 21:59:57.682704 22326 webserver.cc:460] Webserver started at http://127.2.70.195:33303/ using document root <none> and password file <none>
I20260925 21:59:57.682989 22326 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:57.683046 22326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:57.683173 22326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:57.684283 22326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/data/instance:
uuid: "1247e96e73f44ea6873a8e4aa349a5bb"
format_stamp: "Formatted at 2026-09-25 21:59:57 on dist-test-slave-2g0f"
server_key: "339de499c2befac605ff181a9a173d34"
server_key_iv: "e29ffe27af5906d7d779fbbd93daf008"
server_key_version: "encryptionkey@0"
I20260925 21:59:57.684607 22326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance:
uuid: "1247e96e73f44ea6873a8e4aa349a5bb"
format_stamp: "Formatted at 2026-09-25 21:59:57 on dist-test-slave-2g0f"
server_key: "339de499c2befac605ff181a9a173d34"
server_key_iv: "e29ffe27af5906d7d779fbbd93daf008"
server_key_version: "encryptionkey@0"
I20260925 21:59:57.685793 22326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 21:59:57.686442 22340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:57.686590 22326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 21:59:57.686667 22326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/wal
uuid: "1247e96e73f44ea6873a8e4aa349a5bb"
format_stamp: "Formatted at 2026-09-25 21:59:57 on dist-test-slave-2g0f"
server_key: "339de499c2befac605ff181a9a173d34"
server_key_iv: "e29ffe27af5906d7d779fbbd93daf008"
server_key_version: "encryptionkey@0"
I20260925 21:59:57.686720 22326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:57.694190 22326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:57.694471 22326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:57.694574 22326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:57.694756 22326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 21:59:57.695040 22326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 21:59:57.695070 22326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:57.695091 22326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 21:59:57.695107 22326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:57.700469 22326 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.195:41637
I20260925 21:59:57.700544 22453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.195:41637 every 8 connection(s)
I20260925 21:59:57.700825 22326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/data/info.pb
I20260925 21:59:57.706252 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.701546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.195:55359 (local address 127.2.70.254:34617)
0925 21:59:57.701611 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.701612 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.701807 (+ 195us) server_negotiation.cc:407] Connection header received
0925 21:59:57.702303 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.702304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.702313 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.702371 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:57.703045 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.703557 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.704727 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.704922 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.704981 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.705959 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:57.705960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:57.705961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:57.705966 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:57.705981 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:57.705985 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:57.706036 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.706129 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.706155 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 21:59:57.706403 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.701519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34617 (local address 127.2.70.195:55359)
0925 21:59:57.701694 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:57.701734 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:57.702222 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:57.702439 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:57.702443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:57.702451 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 21:59:57.702951 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.702955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.703691 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.703692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.704611 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.704614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.705024 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.705025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.705162 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.705760 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 21:59:57.705761 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 21:59:57.705792 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 21:59:57.705817 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 21:59:57.706037 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 21:59:57.706041 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 21:59:57.706043 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 21:59:57.706098 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 21:59:57.706185 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260925 21:59:57.706775 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 22326
I20260925 21:59:57.706897 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/wal/instance
I20260925 21:59:57.706923 22454 heartbeater.cc:347] Connected to a master server at 127.2.70.254:34617
I20260925 21:59:57.707026 22454 heartbeater.cc:464] Registering TS with master...
I20260925 21:59:57.707134 2331 external_mini_cluster.cc:1428] Setting key 19b7ceb3e894d0ec2fd53230b03d171e
I20260925 21:59:57.707221 22454 heartbeater.cc:511] Master 127.2.70.254:34617 requested a full tablet report, sending...
I20260925 21:59:57.707669 22004 ts_manager.cc:194] Registered new tserver with Master: 1247e96e73f44ea6873a8e4aa349a5bb (127.2.70.195:41637)
I20260925 21:59:57.708052 2331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 21:59:57.708145 22004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.195:55359
I20260925 21:59:57.712363 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.709512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.2.70.254:34617)
0925 21:59:57.709588 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.709589 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.709645 (+ 56us) server_negotiation.cc:407] Connection header received
0925 21:59:57.709705 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.709706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.709717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.709764 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:57.709918 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.710416 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.711197 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.711380 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.711434 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.712134 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:57.712135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:57.712136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 21:59:57.712140 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:57.712155 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 21:59:57.712159 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:57.712203 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.712257 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.712282 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 21:59:57.714048 22004 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 21:59:57.715543 22004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 21:59:57.721814 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.718001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.2.70.195:41637)
0925 21:59:57.718310 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.718315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.718324 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:59:57.718365 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.718368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.718384 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.718487 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:57.718670 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.719722 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.720887 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.721421 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.721486 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.721593 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.721674 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.721715 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":79,"threads_started":1}
I20260925 21:59:57.721805 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.717885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:41637 (local address 127.0.0.1:44934)
0925 21:59:57.718064 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:57.718118 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:57.718184 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:57.718496 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:57.718499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:57.718501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:57.718583 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.718586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.719785 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.719786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.720805 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.720808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.721498 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.721499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.721583 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.721597 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 21:59:57.721661 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 21:59:57.721715 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":64,"threads_started":1}
I20260925 21:59:57.722469 22388 tablet_service.cc:1467] Processing CreateTablet for tablet 7d177931d289434b933b901cf0f6da88 (DEFAULT_TABLE table=test-table [id=760f1187c2bb48e3bc3c0d7419c3a227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:57.722834 22388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d177931d289434b933b901cf0f6da88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:57.723217 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.718197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:37365 (local address 127.0.0.1:46640)
0925 21:59:57.718586 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:57.718595 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:57.719099 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:57.719305 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:57.719306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:57.719308 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:57.719342 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.719344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.720681 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.720682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.721786 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.721789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.722968 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.722969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.723030 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.723034 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:59:57.723082 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 21:59:57.723130 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":82,"threads_started":1}
I20260925 21:59:57.723408 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.717956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:46697 (local address 127.0.0.1:45954)
0925 21:59:57.718312 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:57.718354 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:57.718411 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:57.718971 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:57.718972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:57.718974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:57.719017 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.719019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.720496 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.720498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.721813 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.721815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.723120 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.723121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.723186 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.723189 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:57.723238 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 21:59:57.723272 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":3,"thread_start_us":129,"threads_started":1}
I20260925 21:59:57.723469 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.718312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.2.70.194:37365)
0925 21:59:57.718493 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.718497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.718938 (+ 441us) server_negotiation.cc:407] Connection header received
0925 21:59:57.719197 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.719200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.719216 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.719310 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:57.719396 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.720604 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.722029 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.722768 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.722840 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.722925 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.723333 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.723380 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260925 21:59:57.723752 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.717984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.2.70.193:46697)
0925 21:59:57.718738 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.718743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.718752 (+ 9us) server_negotiation.cc:407] Connection header received
0925 21:59:57.718789 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.718791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.718806 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.718875 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:57.719126 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.720384 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.721902 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.722924 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.722998 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.723085 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.723261 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.723673 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":106,"threads_started":1}
I20260925 21:59:57.724130 22257 tablet_service.cc:1467] Processing CreateTablet for tablet 7d177931d289434b933b901cf0f6da88 (DEFAULT_TABLE table=test-table [id=760f1187c2bb48e3bc3c0d7419c3a227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:57.724460 22257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d177931d289434b933b901cf0f6da88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:57.724556 22126 tablet_service.cc:1467] Processing CreateTablet for tablet 7d177931d289434b933b901cf0f6da88 (DEFAULT_TABLE table=test-table [id=760f1187c2bb48e3bc3c0d7419c3a227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 21:59:57.724900 22126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d177931d289434b933b901cf0f6da88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:57.726323 22468 tablet_bootstrap.cc:492] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb: Bootstrap starting.
I20260925 21:59:57.727049 22468 tablet_bootstrap.cc:654] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:57.727371 22468 log.cc:824] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:57.727392 22469 tablet_bootstrap.cc:492] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8: Bootstrap starting.
I20260925 21:59:57.727981 22471 tablet_bootstrap.cc:492] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e: Bootstrap starting.
I20260925 21:59:57.728106 22469 tablet_bootstrap.cc:654] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:57.728246 22468 tablet_bootstrap.cc:492] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb: No bootstrap required, opened a new log
I20260925 21:59:57.728322 22468 ts_tablet_manager.cc:1397] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:59:57.728402 22469 log.cc:824] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:57.728652 22471 tablet_bootstrap.cc:654] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:57.728991 22469 tablet_bootstrap.cc:492] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8: No bootstrap required, opened a new log
I20260925 21:59:57.728971 22471 log.cc:824] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:57.729048 22469 ts_tablet_manager.cc:1397] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:59:57.729681 22471 tablet_bootstrap.cc:492] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e: No bootstrap required, opened a new log
I20260925 21:59:57.729748 22471 ts_tablet_manager.cc:1397] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 21:59:57.730851 22469 raft_consensus.cc:348] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb9b6fd76ff648febfc13a6ef421966e" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46697 } } peers { permanent_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37365 } } peers { permanent_uuid: "1247e96e73f44ea6873a8e4aa349a5bb" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 41637 } }
I20260925 21:59:57.730978 22469 raft_consensus.cc:374] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:57.731009 22469 raft_consensus.cc:729] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6433d546b0cb4c5b9d9d71bcb3a3d8d8, State: Initialized, Role: FOLLOWER
I20260925 21:59:57.731024 22468 raft_consensus.cc:348] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb9b6fd76ff648febfc13a6ef421966e" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46697 } } peers { permanent_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37365 } } peers { permanent_uuid: "1247e96e73f44ea6873a8e4aa349a5bb" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 41637 } }
I20260925 21:59:57.731149 22469 consensus_queue.cc:260] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb9b6fd76ff648febfc13a6ef421966e" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46697 } } peers { permanent_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37365 } } peers { permanent_uuid: "1247e96e73f44ea6873a8e4aa349a5bb" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 41637 } }
I20260925 21:59:57.731180 22468 raft_consensus.cc:374] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:57.731218 22468 raft_consensus.cc:729] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1247e96e73f44ea6873a8e4aa349a5bb, State: Initialized, Role: FOLLOWER
I20260925 21:59:57.731323 22468 consensus_queue.cc:260] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb9b6fd76ff648febfc13a6ef421966e" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46697 } } peers { permanent_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37365 } } peers { permanent_uuid: "1247e96e73f44ea6873a8e4aa349a5bb" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 41637 } }
I20260925 21:59:57.731439 22469 ts_tablet_manager.cc:1428] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 21:59:57.731508 22323 heartbeater.cc:503] Master 127.2.70.254:34617 was elected leader, sending a full tablet report...
I20260925 21:59:57.731568 22468 ts_tablet_manager.cc:1428] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 21:59:57.731626 22454 heartbeater.cc:503] Master 127.2.70.254:34617 was elected leader, sending a full tablet report...
I20260925 21:59:57.732316 22471 raft_consensus.cc:348] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb9b6fd76ff648febfc13a6ef421966e" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46697 } } peers { permanent_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37365 } } peers { permanent_uuid: "1247e96e73f44ea6873a8e4aa349a5bb" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 41637 } }
I20260925 21:59:57.732470 22471 raft_consensus.cc:374] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:57.732514 22471 raft_consensus.cc:729] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9b6fd76ff648febfc13a6ef421966e, State: Initialized, Role: FOLLOWER
I20260925 21:59:57.732651 22471 consensus_queue.cc:260] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb9b6fd76ff648febfc13a6ef421966e" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46697 } } peers { permanent_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37365 } } peers { permanent_uuid: "1247e96e73f44ea6873a8e4aa349a5bb" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 41637 } }
I20260925 21:59:57.732878 22471 ts_tablet_manager.cc:1428] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 21:59:57.732933 22192 heartbeater.cc:503] Master 127.2.70.254:34617 was elected leader, sending a full tablet report...
I20260925 21:59:57.742004 22476 raft_consensus.cc:482] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 21:59:57.742154 22476 raft_consensus.cc:504] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb9b6fd76ff648febfc13a6ef421966e" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46697 } } peers { permanent_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37365 } } peers { permanent_uuid: "1247e96e73f44ea6873a8e4aa349a5bb" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 41637 } }
I20260925 21:59:57.743167 22476 leader_election.cc:288] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6433d546b0cb4c5b9d9d71bcb3a3d8d8 (127.2.70.194:37365), 1247e96e73f44ea6873a8e4aa349a5bb (127.2.70.195:41637)
I20260925 21:59:57.746083 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.742842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:50033 (local address 127.2.70.194:37365)
0925 21:59:57.742884 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.742886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.742897 (+ 11us) server_negotiation.cc:407] Connection header received
0925 21:59:57.743010 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.743012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.743020 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.743081 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:57.743230 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.743705 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.745159 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.745685 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.745734 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.745742 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.745964 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.746021 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 21:59:57.746078 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.742660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.195:41637 (local address 127.2.70.193:55757)
0925 21:59:57.742707 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:57.742717 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:57.742755 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:57.743007 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:57.743008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:57.743010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:57.743093 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.743095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.744012 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.744013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.745056 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.745059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.745788 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.745789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.745890 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.745893 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 21:59:57.745941 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 21:59:57.745981 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 21:59:57.746315 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.742829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.193:55757 (local address 127.2.70.195:41637)
0925 21:59:57.742872 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.742873 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.742880 (+ 7us) server_negotiation.cc:407] Connection header received
0925 21:59:57.742906 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.742907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.742915 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.742958 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 21:59:57.743158 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.743910 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.745143 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.745692 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.745762 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.745770 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.746203 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.746248 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 21:59:57.746086 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.742684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.194:37365 (local address 127.2.70.193:50033)
0925 21:59:57.742846 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:57.742855 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 21:59:57.742897 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 21:59:57.743081 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 21:59:57.743081 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 21:59:57.743083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 21:59:57.743150 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.743152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.743819 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.743820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.745055 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 21:59:57.745058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.745809 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.745810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 21:59:57.745873 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.745877 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 21:59:57.745925 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 21:59:57.745965 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260925 21:59:57.746793 22277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d177931d289434b933b901cf0f6da88" candidate_uuid: "eb9b6fd76ff648febfc13a6ef421966e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8" is_pre_election: true
I20260925 21:59:57.746958 22277 raft_consensus.cc:2454] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb9b6fd76ff648febfc13a6ef421966e in term 0.
I20260925 21:59:57.747232 22082 leader_election.cc:302] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6433d546b0cb4c5b9d9d71bcb3a3d8d8, eb9b6fd76ff648febfc13a6ef421966e; no voters:
I20260925 21:59:57.747359 22476 raft_consensus.cc:2781] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 21:59:57.747406 22476 raft_consensus.cc:482] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 21:59:57.747429 22476 raft_consensus.cc:3037] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:57.748149 22408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d177931d289434b933b901cf0f6da88" candidate_uuid: "eb9b6fd76ff648febfc13a6ef421966e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1247e96e73f44ea6873a8e4aa349a5bb" is_pre_election: true
I20260925 21:59:57.748296 22408 raft_consensus.cc:2454] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb9b6fd76ff648febfc13a6ef421966e in term 0.
I20260925 21:59:57.748353 22476 raft_consensus.cc:504] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb9b6fd76ff648febfc13a6ef421966e" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46697 } } peers { permanent_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37365 } } peers { permanent_uuid: "1247e96e73f44ea6873a8e4aa349a5bb" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 41637 } }
I20260925 21:59:57.748487 22476 leader_election.cc:288] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [CANDIDATE]: Term 1 election: Requested vote from peers 6433d546b0cb4c5b9d9d71bcb3a3d8d8 (127.2.70.194:37365), 1247e96e73f44ea6873a8e4aa349a5bb (127.2.70.195:41637)
I20260925 21:59:57.748608 22277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d177931d289434b933b901cf0f6da88" candidate_uuid: "eb9b6fd76ff648febfc13a6ef421966e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8"
I20260925 21:59:57.748678 22277 raft_consensus.cc:3037] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:57.748802 22408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d177931d289434b933b901cf0f6da88" candidate_uuid: "eb9b6fd76ff648febfc13a6ef421966e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1247e96e73f44ea6873a8e4aa349a5bb"
I20260925 21:59:57.748893 22408 raft_consensus.cc:3037] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:57.749511 22277 raft_consensus.cc:2454] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb9b6fd76ff648febfc13a6ef421966e in term 1.
I20260925 21:59:57.749758 22408 raft_consensus.cc:2454] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb9b6fd76ff648febfc13a6ef421966e in term 1.
I20260925 21:59:57.749804 22082 leader_election.cc:302] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6433d546b0cb4c5b9d9d71bcb3a3d8d8, eb9b6fd76ff648febfc13a6ef421966e; no voters:
I20260925 21:59:57.749888 22476 raft_consensus.cc:2781] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:57.750015 22476 raft_consensus.cc:686] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [term 1 LEADER]: Becoming Leader. State: Replica: eb9b6fd76ff648febfc13a6ef421966e, State: Running, Role: LEADER
I20260925 21:59:57.750097 22476 consensus_queue.cc:237] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb9b6fd76ff648febfc13a6ef421966e" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46697 } } peers { permanent_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37365 } } peers { permanent_uuid: "1247e96e73f44ea6873a8e4aa349a5bb" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 41637 } }
I20260925 21:59:57.750901 22003 catalog_manager.cc:5495] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e reported cstate change: term changed from 0 to 1, leader changed from <none> to eb9b6fd76ff648febfc13a6ef421966e (127.2.70.193). New cstate: current_term: 1 leader_uuid: "eb9b6fd76ff648febfc13a6ef421966e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb9b6fd76ff648febfc13a6ef421966e" member_type: VOTER last_known_addr { host: "127.2.70.193" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1247e96e73f44ea6873a8e4aa349a5bb" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 41637 } health_report { overall_health: UNKNOWN } } }
I20260925 21:59:57.757061 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.754689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.2.70.193:46697)
0925 21:59:57.754744 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:57.754746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:57.754754 (+ 8us) server_negotiation.cc:407] Connection header received
0925 21:59:57.754785 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:57.754785 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:57.754805 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:57.754872 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 21:59:57.755014 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.755484 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.756361 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:57.756571 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:57.756625 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:57.756759 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 21:59:57.756862 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 21:59:57.756935 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:57.756952 (+ 17us) server_negotiation.cc:299] Negotiation successful
0925 21:59:57.756971 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 21:59:57.758769 22277 raft_consensus.cc:1261] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8 [term 1 FOLLOWER]: Refusing update from remote peer eb9b6fd76ff648febfc13a6ef421966e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:57.758769 22408 raft_consensus.cc:1261] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb [term 1 FOLLOWER]: Refusing update from remote peer eb9b6fd76ff648febfc13a6ef421966e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 21:59:57.759151 22476 consensus_queue.cc:1035] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1247e96e73f44ea6873a8e4aa349a5bb" member_type: VOTER last_known_addr { host: "127.2.70.195" port: 41637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:57.759295 22478 consensus_queue.cc:1035] T 7d177931d289434b933b901cf0f6da88 P eb9b6fd76ff648febfc13a6ef421966e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6433d546b0cb4c5b9d9d71bcb3a3d8d8" member_type: VOTER last_known_addr { host: "127.2.70.194" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 21:59:57.761507 22482 mvcc.cc:205] Tried to move back new op lower bound from 7333370256417476608 to 7333370256384987136. Current Snapshot: MvccSnapshot[applied={T|T < 7333370256417476608}]
I20260925 21:59:57.762490 22484 mvcc.cc:205] Tried to move back new op lower bound from 7333370256417476608 to 7333370256384987136. Current Snapshot: MvccSnapshot[applied={T|T < 7333370256417476608 or (T in {7333370256417476608})}]
I20260925 21:59:57.763814 22004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:57.763927 22004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 21:59:57.764634 22004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18538e883e894db581e2b28279b8ee1d: Sending DeleteTablet for 3 replicas of tablet 7d177931d289434b933b901cf0f6da88
I20260925 21:59:57.764940 22388 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d177931d289434b933b901cf0f6da88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:57 UTC) from {username='slave'} at 127.0.0.1:44934
I20260925 21:59:57.764990 22126 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d177931d289434b933b901cf0f6da88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:57 UTC) from {username='slave'} at 127.0.0.1:45954
I20260925 21:59:57.765178 22490 tablet_replica.cc:330] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb: stopping tablet replica
I20260925 21:59:57.765178 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 22064
I20260925 21:59:57.765200 22257 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d177931d289434b933b901cf0f6da88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 21:59:57 UTC) from {username='slave'} at 127.0.0.1:46640
I20260925 21:59:57.765308 22490 raft_consensus.cc:2229] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:59:57.765427 22490 raft_consensus.cc:2258] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:57.765877 22490 ts_tablet_manager.cc:1905] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:57.767021 22492 tablet_replica.cc:330] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8: stopping tablet replica
I20260925 21:59:57.767138 22492 raft_consensus.cc:2229] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 21:59:57.767288 22492 raft_consensus.cc:2258] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 21:59:57.767311 22490 ts_tablet_manager.cc:1918] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:57.767436 22490 log.cc:1196] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-2/wal/wals/7d177931d289434b933b901cf0f6da88
I20260925 21:59:57.767730 22492 ts_tablet_manager.cc:1905] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 21:59:57.767815 22490 ts_tablet_manager.cc:1939] T 7d177931d289434b933b901cf0f6da88 P 1247e96e73f44ea6873a8e4aa349a5bb: Deleting consensus metadata
I20260925 21:59:57.768252 21989 catalog_manager.cc:4841] TS 1247e96e73f44ea6873a8e4aa349a5bb (127.2.70.195:41637): tablet 7d177931d289434b933b901cf0f6da88 (table test-table [id=760f1187c2bb48e3bc3c0d7419c3a227]) successfully deleted
I20260925 21:59:57.769233 22492 ts_tablet_manager.cc:1918] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 21:59:57.769290 22492 log.cc:1196] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8: Deleting WAL directory at /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790373357410105-2331-0/minicluster-data/ts-1/wal/wals/7d177931d289434b933b901cf0f6da88
I20260925 21:59:57.769512 22492 ts_tablet_manager.cc:1939] T 7d177931d289434b933b901cf0f6da88 P 6433d546b0cb4c5b9d9d71bcb3a3d8d8: Deleting consensus metadata
I20260925 21:59:57.769870 21991 catalog_manager.cc:4841] TS 6433d546b0cb4c5b9d9d71bcb3a3d8d8 (127.2.70.194:37365): tablet 7d177931d289434b933b901cf0f6da88 (table test-table [id=760f1187c2bb48e3bc3c0d7419c3a227]) successfully deleted
W20260925 21:59:57.770417 21991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.193:46697 (error 108)
I20260925 21:59:57.770944 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 22195
I20260925 21:59:57.770893 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:57.770774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.193:46697 (local address 127.0.0.1:45968)
0925 21:59:57.770824 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 21:59:57.770859 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.193:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 21:59:57.771631 21991 catalog_manager.cc:4584] TS eb9b6fd76ff648febfc13a6ef421966e (127.2.70.193:46697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d177931d289434b933b901cf0f6da88: Network error: Client connection negotiation failed: client connection to 127.2.70.193:46697: connect: Connection refused (error 111)
I20260925 21:59:57.775251 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 22326
I20260925 21:59:57.779376 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 21973
2026-09-25T21:59:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (610 ms)
[----------] 6 tests from EncryptionPolicyTest (8052 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:59:57 dist-test-slave-2g0f krb5kdc[22496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:59:57 dist-test-slave-2g0f krb5kdc[22496](info): set up 2 sockets
Sep 25 21:59:57 dist-test-slave-2g0f krb5kdc[22496](info): commencing operation
krb5kdc: starting...
W20260925 21:59:58.735530 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 21:59:58 dist-test-slave-2g0f krb5kdc[22496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T21:59:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T21:59:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 21:59:58.850793 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 21:59:58.931707 22509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 21:59:58.931953 22509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 21:59:58.931977 22509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 21:59:58.933156 22509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 21:59:58.933195 22509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 21:59:58.933208 22509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 21:59:58.933231 22509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 21:59:58.933254 22509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 21:59:58.934743 22509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45161
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 21:59:58.934980 22509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 21:59:58.935189 22509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 21:59:58.939802 22515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:58.939913 22514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 21:59:58.939810 22517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 21:59:58.939975 22509 server_base.cc:1034] running on GCE node
I20260925 21:59:58.940239 22509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 21:59:58.940521 22509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 21:59:58.941676 22509 hybrid_clock.cc:648] HybridClock initialized: now 1790373598941659 us; error 28 us; skew 500 ppm
Sep 25 21:59:58 dist-test-slave-2g0f krb5kdc[22496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 21:59:58.944070 22509 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 21:59:58.944486 22509 webserver.cc:460] Webserver started at http://127.0.0.1:41601/ using document root <none> and password file <none>
I20260925 21:59:58.944705 22509 fs_manager.cc:359] Metadata directory not provided
I20260925 21:59:58.944753 22509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 21:59:58.944875 22509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 21:59:58.945919 22509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "b413dc1a0d5648f894a06d00d5c49dc5"
format_stamp: "Formatted at 2026-09-25 21:59:58 on dist-test-slave-2g0f"
server_key: "adb655a9d4f966323a321fda15adabd8"
server_key_iv: "50d1d97b5957f613e9331899c1409dd2"
server_key_version: "encryptionkey@0"
I20260925 21:59:58.946235 22509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "b413dc1a0d5648f894a06d00d5c49dc5"
format_stamp: "Formatted at 2026-09-25 21:59:58 on dist-test-slave-2g0f"
server_key: "adb655a9d4f966323a321fda15adabd8"
server_key_iv: "50d1d97b5957f613e9331899c1409dd2"
server_key_version: "encryptionkey@0"
I20260925 21:59:58.947431 22509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 21:59:58.948143 22524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:58.948300 22509 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 21:59:58.948382 22509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "b413dc1a0d5648f894a06d00d5c49dc5"
format_stamp: "Formatted at 2026-09-25 21:59:58 on dist-test-slave-2g0f"
server_key: "adb655a9d4f966323a321fda15adabd8"
server_key_iv: "50d1d97b5957f613e9331899c1409dd2"
server_key_version: "encryptionkey@0"
I20260925 21:59:58.948443 22509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 21:59:58.966742 22509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 21:59:58.968688 22509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 21:59:58.968829 22509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 21:59:58.972610 22509 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45161
I20260925 21:59:58.972656 22576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45161 every 8 connection(s)
I20260925 21:59:58.972976 22509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 21:59:58.973798 22577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 21:59:58.975265 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 22509
I20260925 21:59:58.975368 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 21:59:58.975561 2331 external_mini_cluster.cc:1428] Setting key 879c7f83fed34c18101835f03f8781f2
I20260925 21:59:58.976421 22577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5: Bootstrap starting.
I20260925 21:59:58.976970 22577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5: Neither blocks nor log segments found. Creating new log.
I20260925 21:59:58.977274 22577 log.cc:824] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5: Log is configured to *not* fsync() on all Append() calls
I20260925 21:59:58.977949 22577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5: No bootstrap required, opened a new log
I20260925 21:59:58.979978 22577 raft_consensus.cc:348] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b413dc1a0d5648f894a06d00d5c49dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45161 } }
I20260925 21:59:58.980114 22577 raft_consensus.cc:374] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 21:59:58.980136 22577 raft_consensus.cc:729] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b413dc1a0d5648f894a06d00d5c49dc5, State: Initialized, Role: FOLLOWER
I20260925 21:59:58.980260 22577 consensus_queue.cc:260] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b413dc1a0d5648f894a06d00d5c49dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45161 } }
I20260925 21:59:58.980335 22577 raft_consensus.cc:388] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 21:59:58.980366 22577 raft_consensus.cc:482] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 21:59:58.980410 22577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 21:59:58.981037 22577 raft_consensus.cc:504] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b413dc1a0d5648f894a06d00d5c49dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45161 } }
I20260925 21:59:58.981137 22577 leader_election.cc:302] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b413dc1a0d5648f894a06d00d5c49dc5; no voters:
I20260925 21:59:58.981289 22577 leader_election.cc:288] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 21:59:58.981351 22582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 21:59:58.981477 22582 raft_consensus.cc:686] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [term 1 LEADER]: Becoming Leader. State: Replica: b413dc1a0d5648f894a06d00d5c49dc5, State: Running, Role: LEADER
I20260925 21:59:58.981554 22582 consensus_queue.cc:237] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b413dc1a0d5648f894a06d00d5c49dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45161 } }
I20260925 21:59:58.981557 22577 sys_catalog.cc:564] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 21:59:58 dist-test-slave-2g0f krb5kdc[22496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 21:59:58.982775 22584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b413dc1a0d5648f894a06d00d5c49dc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b413dc1a0d5648f894a06d00d5c49dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45161 } } }
I20260925 21:59:58.982859 22584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [sys.catalog]: This master's current role is: LEADER
I20260925 21:59:58.982935 22583 sys_catalog.cc:455] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b413dc1a0d5648f894a06d00d5c49dc5. Latest consensus state: current_term: 1 leader_uuid: "b413dc1a0d5648f894a06d00d5c49dc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b413dc1a0d5648f894a06d00d5c49dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45161 } } }
I20260925 21:59:58.983060 22583 sys_catalog.cc:458] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5 [sys.catalog]: This master's current role is: LEADER
W20260925 21:59:58.983429 22596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 21:59:58.983480 22596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 21:59:58.983543 22597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 21:59:58.983685 22597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 21:59:58.985683 22597 catalog_manager.cc:1347] Generated new cluster ID: 58e7e4cb547346b1bc99cd6608d2325e
I20260925 21:59:58.985733 22597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 21:59:58.986620 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:58.976170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59927 (local address 127.0.0.1:45161)
0925 21:59:58.976341 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:58.976349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:58.976376 (+ 27us) server_negotiation.cc:407] Connection header received
0925 21:59:58.976858 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:58.976865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:58.976917 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:58.977051 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:58.977749 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:58.978667 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:58.979730 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:58.980245 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:58.980370 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:58.982874 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:58.982879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:58.982879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:58.982891 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:58.984665 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:58.985082 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:58.985085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:58.985085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:58.985191 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:58.985409 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:58.985410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:58.985410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:58.985711 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:58.985862 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:58.986311 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 21:59:58.986425 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
I20260925 21:59:58.987507 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 21:59:58.992748 22597 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 21:59:58.993340 22597 catalog_manager.cc:1504] Loading token signing keys...
I20260925 21:59:58.995899 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 21:59:58.988965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43565 (local address 127.0.0.1:45161)
0925 21:59:58.989005 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 21:59:58.989007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 21:59:58.989100 (+ 93us) server_negotiation.cc:407] Connection header received
0925 21:59:58.989157 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 21:59:58.989158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 21:59:58.989169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 21:59:58.989221 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 21:59:58.989746 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:58.990497 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:58.991452 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 21:59:58.991774 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 21:59:58.991884 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 21:59:58.993209 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 21:59:58.993211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 21:59:58.993211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 21:59:58.993217 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 21:59:58.994770 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:58.995159 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:58.995160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:58.995160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:58.995217 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 21:59:58.995398 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 21:59:58.995399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 21:59:58.995399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 21:59:58.995517 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 21:59:58.995618 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 21:59:58.995722 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 21:59:58.995782 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 21:59:58.998988 22597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b413dc1a0d5648f894a06d00d5c49dc5: Generated new TSK 0
I20260925 21:59:58.999153 22597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 21:59:59.007328 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 22509
W20260925 21:59:59.011000 22494 connection.cc:535] client connection to 127.0.0.1:45161 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45161): Connection reset by peer (error 104)
2026-09-25T21:59:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 21:59:59 dist-test-slave-2g0f krb5kdc[22608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 21:59:59 dist-test-slave-2g0f krb5kdc[22608](info): set up 2 sockets
Sep 25 21:59:59 dist-test-slave-2g0f krb5kdc[22608](info): commencing operation
krb5kdc: starting...
W20260925 21:59:59.951545 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:00 dist-test-slave-2g0f krb5kdc[22608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:00:00.071432 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:00.153463 22621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:00.153681 22621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:00.153705 22621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:00.154886 22621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:00.154955 22621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:00.154973 22621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:00.154999 22621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:00.155017 22621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:00.156599 22621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40175
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:00.156836 22621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:00.157049 22621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:00.161787 22626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:00.161793 22627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:00.161845 22629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:00.161945 22621 server_base.cc:1034] running on GCE node
I20260925 22:00:00.162324 22621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:00.162607 22621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:00.163774 22621 hybrid_clock.cc:648] HybridClock initialized: now 1790373600163757 us; error 26 us; skew 500 ppm
Sep 25 22:00:00 dist-test-slave-2g0f krb5kdc[22608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:00.166095 22621 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:00:00.166484 22621 webserver.cc:460] Webserver started at http://127.0.0.1:37813/ using document root <none> and password file <none>
I20260925 22:00:00.166695 22621 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:00.166742 22621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:00.166862 22621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:00.167948 22621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "24065af833ad4c70ae5789ed3ad4c25f"
format_stamp: "Formatted at 2026-09-25 22:00:00 on dist-test-slave-2g0f"
server_key: "945eddcb96646eca1c01c20598d834e5"
server_key_iv: "695efeb04d783916364fe6ea7701dace"
server_key_version: "encryptionkey@0"
I20260925 22:00:00.168273 22621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "24065af833ad4c70ae5789ed3ad4c25f"
format_stamp: "Formatted at 2026-09-25 22:00:00 on dist-test-slave-2g0f"
server_key: "945eddcb96646eca1c01c20598d834e5"
server_key_iv: "695efeb04d783916364fe6ea7701dace"
server_key_version: "encryptionkey@0"
I20260925 22:00:00.169492 22621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:00:00.170239 22636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:00.170398 22621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:00:00.170480 22621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "24065af833ad4c70ae5789ed3ad4c25f"
format_stamp: "Formatted at 2026-09-25 22:00:00 on dist-test-slave-2g0f"
server_key: "945eddcb96646eca1c01c20598d834e5"
server_key_iv: "695efeb04d783916364fe6ea7701dace"
server_key_version: "encryptionkey@0"
I20260925 22:00:00.170539 22621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:00.183455 22621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:00.185467 22621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:00.185593 22621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:00.189338 22621 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40175
I20260925 22:00:00.189381 22688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40175 every 8 connection(s)
I20260925 22:00:00.189700 22621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:00.190670 22689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:00.193188 22689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f: Bootstrap starting.
I20260925 22:00:00.193698 22689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:00.193923 22689 log.cc:824] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:00.194557 22689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f: No bootstrap required, opened a new log
I20260925 22:00:00.196161 22689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24065af833ad4c70ae5789ed3ad4c25f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40175 } }
I20260925 22:00:00.196264 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 22621
I20260925 22:00:00.196293 22689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:00.196316 22689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24065af833ad4c70ae5789ed3ad4c25f, State: Initialized, Role: FOLLOWER
I20260925 22:00:00.196369 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:00.196481 22689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24065af833ad4c70ae5789ed3ad4c25f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40175 } }
I20260925 22:00:00.196554 22689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:00.196563 2331 external_mini_cluster.cc:1428] Setting key be74f7e1bc4e44e0362be82fb2f21ecf
I20260925 22:00:00.196576 22689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:00.196615 22689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:00.197598 22689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24065af833ad4c70ae5789ed3ad4c25f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40175 } }
I20260925 22:00:00.197703 22689 leader_election.cc:302] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24065af833ad4c70ae5789ed3ad4c25f; no voters:
I20260925 22:00:00.197860 22689 leader_election.cc:288] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:00.197912 22694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:00.198065 22694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [term 1 LEADER]: Becoming Leader. State: Replica: 24065af833ad4c70ae5789ed3ad4c25f, State: Running, Role: LEADER
I20260925 22:00:00.198097 22689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:00.198163 22694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24065af833ad4c70ae5789ed3ad4c25f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40175 } }
I20260925 22:00:00.198511 22694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24065af833ad4c70ae5789ed3ad4c25f. Latest consensus state: current_term: 1 leader_uuid: "24065af833ad4c70ae5789ed3ad4c25f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24065af833ad4c70ae5789ed3ad4c25f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40175 } } }
I20260925 22:00:00.198593 22694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:00.198748 22696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24065af833ad4c70ae5789ed3ad4c25f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24065af833ad4c70ae5789ed3ad4c25f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40175 } } }
I20260925 22:00:00.198825 22696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:00.198962 22702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:00.199493 22702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:00.201628 22702 catalog_manager.cc:1347] Generated new cluster ID: 7391961f55ab43d9aa191c0437b3c134
I20260925 22:00:00.201680 22702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:00.202373 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:00.197024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59989 (local address 127.0.0.1:40175)
0925 22:00:00.197225 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:00.197236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:00.197273 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:00:00.197390 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:00.197397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:00.197452 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:00.197604 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:00.197845 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:00.199303 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:00.200349 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:00.200799 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:00.200898 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:00.201599 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:00.201603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:00.201603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:00.201797 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:00.201836 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:00.201850 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:00.201933 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:00.202064 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 22:00:00.202174 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260925 22:00:00.203219 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:00.207991 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:00.204550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38265 (local address 127.0.0.1:40175)
0925 22:00:00.204587 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:00.204588 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:00.204733 (+ 145us) server_negotiation.cc:407] Connection header received
0925 22:00:00.204800 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:00.204800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:00.204810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:00.204859 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:00.204997 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:00.205697 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:00.206647 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:00.206951 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:00.207002 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:00.207719 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:00.207720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:00.207721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:00.207724 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:00.207735 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:00.207739 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:00.207783 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:00.207906 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 22:00:00.207917 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 22:00:00.212172 22702 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:00.212720 22702 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:00.220139 22702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24065af833ad4c70ae5789ed3ad4c25f: Generated new TSK 0
I20260925 22:00:00.220353 22702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:00.227747 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 22621
W20260925 22:00:00.231608 22606 connection.cc:535] client connection to 127.0.0.1:40175 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40175): Connection reset by peer (error 104)
2026-09-25T22:00:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:00 dist-test-slave-2g0f krb5kdc[22720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:00 dist-test-slave-2g0f krb5kdc[22720](info): set up 2 sockets
Sep 25 22:00:00 dist-test-slave-2g0f krb5kdc[22720](info): commencing operation
krb5kdc: starting...
W20260925 22:00:01.183597 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:01 dist-test-slave-2g0f krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:00:01.302803 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:01.382248 22733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:01.382467 22733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:01.382488 22733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:01.383708 22733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:01.383754 22733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:01.383767 22733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:01.383780 22733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:01.383803 22733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:01.385305 22733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43233
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:01.385536 22733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:01.385749 22733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:01.390552 22741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:01.390538 22739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:01.390552 22738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:01.390652 22733 server_base.cc:1034] running on GCE node
I20260925 22:00:01.390995 22733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:01.391249 22733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:01.392381 22733 hybrid_clock.cc:648] HybridClock initialized: now 1790373601392370 us; error 29 us; skew 500 ppm
Sep 25 22:00:01 dist-test-slave-2g0f krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:01.414600 22733 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:00:01.415086 22733 webserver.cc:460] Webserver started at http://127.0.0.1:45183/ using document root <none> and password file <none>
I20260925 22:00:01.415323 22733 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:01.415373 22733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:01.415498 22733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:01.416622 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "00427e0f665d4d4ba3fee4e7e1f4145b"
format_stamp: "Formatted at 2026-09-25 22:00:01 on dist-test-slave-2g0f"
server_key: "3de4e735000b920d272c6b127d6ac766"
server_key_iv: "5f624000c99d8a0cf21da59f884c0008"
server_key_version: "encryptionkey@0"
I20260925 22:00:01.416962 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "00427e0f665d4d4ba3fee4e7e1f4145b"
format_stamp: "Formatted at 2026-09-25 22:00:01 on dist-test-slave-2g0f"
server_key: "3de4e735000b920d272c6b127d6ac766"
server_key_iv: "5f624000c99d8a0cf21da59f884c0008"
server_key_version: "encryptionkey@0"
I20260925 22:00:01.418303 22733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 22:00:01.419121 22748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:01.419291 22733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:00:01.419379 22733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "00427e0f665d4d4ba3fee4e7e1f4145b"
format_stamp: "Formatted at 2026-09-25 22:00:01 on dist-test-slave-2g0f"
server_key: "3de4e735000b920d272c6b127d6ac766"
server_key_iv: "5f624000c99d8a0cf21da59f884c0008"
server_key_version: "encryptionkey@0"
I20260925 22:00:01.419461 22733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:01.422969 22733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:01.425220 22733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:01.425359 22733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:01.429119 22733 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43233
I20260925 22:00:01.429162 22800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43233 every 8 connection(s)
I20260925 22:00:01.429497 22733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:01.430410 22801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:01.432782 22801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b: Bootstrap starting.
I20260925 22:00:01.433342 22801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:01.433597 22801 log.cc:824] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:01.434303 22801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b: No bootstrap required, opened a new log
I20260925 22:00:01.435942 22801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00427e0f665d4d4ba3fee4e7e1f4145b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43233 } }
I20260925 22:00:01.436081 22801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:01.436101 22801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00427e0f665d4d4ba3fee4e7e1f4145b, State: Initialized, Role: FOLLOWER
I20260925 22:00:01.436237 22801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00427e0f665d4d4ba3fee4e7e1f4145b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43233 } }
I20260925 22:00:01.436313 22801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:01.436347 22801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:01.436390 22801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:01.436899 22801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00427e0f665d4d4ba3fee4e7e1f4145b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43233 } }
I20260925 22:00:01.436985 22801 leader_election.cc:302] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00427e0f665d4d4ba3fee4e7e1f4145b; no voters:
I20260925 22:00:01.437132 22801 leader_election.cc:288] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:01.437199 22804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:01.437353 22801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:01.437366 22804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [term 1 LEADER]: Becoming Leader. State: Replica: 00427e0f665d4d4ba3fee4e7e1f4145b, State: Running, Role: LEADER
I20260925 22:00:01.437482 22804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00427e0f665d4d4ba3fee4e7e1f4145b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43233 } }
I20260925 22:00:01.437662 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 22733
I20260925 22:00:01.437767 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:01.438071 2331 external_mini_cluster.cc:1428] Setting key 17cecd1f2a21b8270d0641385740ed4c
I20260925 22:00:01.438292 22806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00427e0f665d4d4ba3fee4e7e1f4145b. Latest consensus state: current_term: 1 leader_uuid: "00427e0f665d4d4ba3fee4e7e1f4145b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00427e0f665d4d4ba3fee4e7e1f4145b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43233 } } }
I20260925 22:00:01.438366 22806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:01.438709 22805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00427e0f665d4d4ba3fee4e7e1f4145b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00427e0f665d4d4ba3fee4e7e1f4145b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43233 } } }
I20260925 22:00:01.438926 22805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b [sys.catalog]: This master's current role is: LEADER
W20260925 22:00:01.439206 22821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:00:01.439261 22821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:00:01.439353 22817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:01.439549 22817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:01.440862 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:01.438703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56613 (local address 127.0.0.1:43233)
0925 22:00:01.439097 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:01.439107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:01.439140 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:00:01.439239 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:01.439245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:01.439274 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:01.439451 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:01.440398 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:01.440400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:01.440401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:01.440404 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:01.440447 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:01.440458 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:01.440549 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:01.440629 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 22:00:01.440682 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":59,"threads_started":1}
I20260925 22:00:01.441013 22817 catalog_manager.cc:1347] Generated new cluster ID: d5bf2bff9290497997155d1b4381655f
I20260925 22:00:01.441087 22817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:01.441812 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:01.444169 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:01.443092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36601 (local address 127.0.0.1:43233)
0925 22:00:01.443129 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:01.443131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:01.443282 (+ 151us) server_negotiation.cc:407] Connection header received
0925 22:00:01.443333 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:01.443334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:01.443338 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:01.443381 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:01.443959 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:01.443960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:01.443960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:01.443965 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:01.443979 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:01.443983 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:01.444036 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:01.444086 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 22:00:01.444096 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 22:00:01.466030 22817 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:01.466228 22817 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:01.469170 22817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00427e0f665d4d4ba3fee4e7e1f4145b: Generated new TSK 0
I20260925 22:00:01.469350 22817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:01.480404 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 22733
2026-09-25T22:00:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:01 dist-test-slave-2g0f krb5kdc[22832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:01 dist-test-slave-2g0f krb5kdc[22832](info): set up 2 sockets
Sep 25 22:00:01 dist-test-slave-2g0f krb5kdc[22832](info): commencing operation
krb5kdc: starting...
W20260925 22:00:02.431701 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:02 dist-test-slave-2g0f krb5kdc[22832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:00:02.547020 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:02.626439 22845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:02.626650 22845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:02.626681 22845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:02.627921 22845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:02.627964 22845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:02.627982 22845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:02.627998 22845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:02.628022 22845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:02.629565 22845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:02.629806 22845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:02.630041 22845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:02.634809 22851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:02.634919 22853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:02.634861 22850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:02.634797 22845 server_base.cc:1034] running on GCE node
I20260925 22:00:02.635332 22845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:02.635600 22845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:02.636739 22845 hybrid_clock.cc:648] HybridClock initialized: now 1790373602636725 us; error 29 us; skew 500 ppm
Sep 25 22:00:02 dist-test-slave-2g0f krb5kdc[22832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:02.639226 22845 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:00:02.639696 22845 webserver.cc:460] Webserver started at http://127.0.0.1:36913/ using document root <none> and password file <none>
I20260925 22:00:02.639968 22845 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:02.640015 22845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:02.640138 22845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:02.641178 22845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "29c670c3b8be4001b74e196478d7b291"
format_stamp: "Formatted at 2026-09-25 22:00:02 on dist-test-slave-2g0f"
server_key: "f1fc9d5d2f9ead716ee7a90beedee494"
server_key_iv: "6c5350ab1abe682b26ed29b442c1914d"
server_key_version: "encryptionkey@0"
I20260925 22:00:02.641563 22845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "29c670c3b8be4001b74e196478d7b291"
format_stamp: "Formatted at 2026-09-25 22:00:02 on dist-test-slave-2g0f"
server_key: "f1fc9d5d2f9ead716ee7a90beedee494"
server_key_iv: "6c5350ab1abe682b26ed29b442c1914d"
server_key_version: "encryptionkey@0"
I20260925 22:00:02.642813 22845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 22:00:02.643555 22860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:02.643711 22845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 22:00:02.643797 22845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "29c670c3b8be4001b74e196478d7b291"
format_stamp: "Formatted at 2026-09-25 22:00:02 on dist-test-slave-2g0f"
server_key: "f1fc9d5d2f9ead716ee7a90beedee494"
server_key_iv: "6c5350ab1abe682b26ed29b442c1914d"
server_key_version: "encryptionkey@0"
I20260925 22:00:02.643894 22845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:02.655136 22845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:02.657198 22845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:02.657308 22845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:02.660909 22845 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34201
I20260925 22:00:02.660960 22912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34201 every 8 connection(s)
I20260925 22:00:02.661262 22845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:02.661692 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 22845
I20260925 22:00:02.661785 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:02.661965 2331 external_mini_cluster.cc:1428] Setting key dbd6b77705b4875b44cd8321c4f4cebe
I20260925 22:00:02.662331 22913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:02.665436 22913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291: Bootstrap starting.
I20260925 22:00:02.666100 22913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:02.666344 22913 log.cc:824] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:02.667047 22913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291: No bootstrap required, opened a new log
Sep 25 22:00:02 dist-test-slave-2g0f krb5kdc[22832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:00:02.668969 22913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c670c3b8be4001b74e196478d7b291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34201 } }
I20260925 22:00:02.669117 22913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:02.669152 22913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c670c3b8be4001b74e196478d7b291, State: Initialized, Role: FOLLOWER
I20260925 22:00:02.669276 22913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c670c3b8be4001b74e196478d7b291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34201 } }
I20260925 22:00:02.669337 22913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:02.669358 22913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:02.669391 22913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:02.669889 22913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c670c3b8be4001b74e196478d7b291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34201 } }
I20260925 22:00:02.669970 22913 leader_election.cc:302] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29c670c3b8be4001b74e196478d7b291; no voters:
I20260925 22:00:02.670111 22913 leader_election.cc:288] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:02.670161 22918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:02.670315 22918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [term 1 LEADER]: Becoming Leader. State: Replica: 29c670c3b8be4001b74e196478d7b291, State: Running, Role: LEADER
I20260925 22:00:02.670364 22913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:02.670400 22918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c670c3b8be4001b74e196478d7b291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34201 } }
I20260925 22:00:02.670761 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29c670c3b8be4001b74e196478d7b291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c670c3b8be4001b74e196478d7b291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34201 } } }
I20260925 22:00:02.670842 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:02.670761 22920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c670c3b8be4001b74e196478d7b291. Latest consensus state: current_term: 1 leader_uuid: "29c670c3b8be4001b74e196478d7b291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c670c3b8be4001b74e196478d7b291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34201 } } }
I20260925 22:00:02.671113 22920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:02.671221 22926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:02.671691 22926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:02.672626 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:02.662445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58833 (local address 127.0.0.1:34201)
0925 22:00:02.662600 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:02.662608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:02.662633 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:00:02.663135 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:02.663142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:02.663190 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:02.663301 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:02.664071 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:02.665003 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:02.666020 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:02.666462 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:02.666541 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:02.668723 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:02.668727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:02.668728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:02.668738 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:02.670606 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:02.671039 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:02.671042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:02.671043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:02.671143 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:02.671345 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:02.671346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:02.671347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:02.671650 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:02.671797 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:02.672331 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 22:00:02.672477 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260925 22:00:02.673106 22935 mvcc.cc:205] Tried to move back new op lower bound from 7333370276547276800 to 7333370276538339328. Current Snapshot: MvccSnapshot[applied={T|T < 7333370276547276800}]
I20260925 22:00:02.673385 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:02.673507 22926 catalog_manager.cc:1347] Generated new cluster ID: e9907a29234f44d8b384c44fc6963f56
I20260925 22:00:02.673552 22926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:02.681582 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:02.674850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38589 (local address 127.0.0.1:34201)
0925 22:00:02.674888 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:02.674890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:02.674992 (+ 102us) server_negotiation.cc:407] Connection header received
0925 22:00:02.675053 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:02.675054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:02.675062 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:02.675105 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:02.675699 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:02.676423 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:02.677436 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:02.677728 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:02.677779 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:02.679074 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:02.679075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:02.679075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:02.679081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:02.680534 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:02.680889 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:02.680890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:02.680890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:02.680937 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:02.681118 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:02.681119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:02.681119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:02.681226 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:02.681300 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:02.681421 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 22:00:02.681495 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 22:00:02.692350 22926 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:02.693118 22926 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:02.700018 22926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29c670c3b8be4001b74e196478d7b291: Generated new TSK 0
I20260925 22:00:02.700243 22926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:02.706228 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 22845
2026-09-25T22:00:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:02 dist-test-slave-2g0f krb5kdc[22944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:02 dist-test-slave-2g0f krb5kdc[22944](info): set up 2 sockets
Sep 25 22:00:02 dist-test-slave-2g0f krb5kdc[22944](info): commencing operation
krb5kdc: starting...
W20260925 22:00:03.659667 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:03 dist-test-slave-2g0f krb5kdc[22944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:00:03.779299 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:03.858286 22957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:03.858487 22957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:03.858510 22957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:03.859606 22957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:03.859637 22957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:03.859648 22957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:03.859660 22957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:03.859670 22957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:03.861085 22957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:03.861271 22957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:03.861441 22957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:03.866133 22962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:03.866149 22963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:03.866202 22957 server_base.cc:1034] running on GCE node
W20260925 22:00:03.866225 22965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:03.866652 22957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:03.866902 22957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:03.868039 22957 hybrid_clock.cc:648] HybridClock initialized: now 1790373603868027 us; error 31 us; skew 500 ppm
Sep 25 22:00:03 dist-test-slave-2g0f krb5kdc[22944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:03.870334 22957 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:00:03.870795 22957 webserver.cc:460] Webserver started at http://127.0.0.1:37481/ using document root <none> and password file <none>
I20260925 22:00:03.871003 22957 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:03.871052 22957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:03.871166 22957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:03.872246 22957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "935424c96fee48fbabc5526c8825bd39"
format_stamp: "Formatted at 2026-09-25 22:00:03 on dist-test-slave-2g0f"
server_key: "999fcaca95a0716bbe56c74f3ce6cc57"
server_key_iv: "7e0a3a67e2af7da478ef64715176d954"
server_key_version: "encryptionkey@0"
I20260925 22:00:03.872640 22957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "935424c96fee48fbabc5526c8825bd39"
format_stamp: "Formatted at 2026-09-25 22:00:03 on dist-test-slave-2g0f"
server_key: "999fcaca95a0716bbe56c74f3ce6cc57"
server_key_iv: "7e0a3a67e2af7da478ef64715176d954"
server_key_version: "encryptionkey@0"
I20260925 22:00:03.873888 22957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 22:00:03.874615 22972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:03.874771 22957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:00:03.874853 22957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "935424c96fee48fbabc5526c8825bd39"
format_stamp: "Formatted at 2026-09-25 22:00:03 on dist-test-slave-2g0f"
server_key: "999fcaca95a0716bbe56c74f3ce6cc57"
server_key_iv: "7e0a3a67e2af7da478ef64715176d954"
server_key_version: "encryptionkey@0"
I20260925 22:00:03.874909 22957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:03.887537 22957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:03.889583 22957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:03.889719 22957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:03.893643 22957 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45847
I20260925 22:00:03.893684 23024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45847 every 8 connection(s)
I20260925 22:00:03.894016 22957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:03.894953 23025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:03.897300 23025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39: Bootstrap starting.
I20260925 22:00:03.897866 23025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:03.898125 23025 log.cc:824] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:03.898864 23025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39: No bootstrap required, opened a new log
I20260925 22:00:03.900593 23025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935424c96fee48fbabc5526c8825bd39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45847 } }
I20260925 22:00:03.900743 23025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:03.900764 23025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935424c96fee48fbabc5526c8825bd39, State: Initialized, Role: FOLLOWER
I20260925 22:00:03.900893 23025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935424c96fee48fbabc5526c8825bd39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45847 } }
I20260925 22:00:03.900964 23025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:03.900987 23025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:03.901036 23025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:03.901615 23025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935424c96fee48fbabc5526c8825bd39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45847 } }
I20260925 22:00:03.901710 23025 leader_election.cc:302] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 935424c96fee48fbabc5526c8825bd39; no voters:
I20260925 22:00:03.901885 23025 leader_election.cc:288] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:03.901923 23028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:03.902123 23025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:03.902129 23028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [term 1 LEADER]: Becoming Leader. State: Replica: 935424c96fee48fbabc5526c8825bd39, State: Running, Role: LEADER
I20260925 22:00:03.902305 23028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935424c96fee48fbabc5526c8825bd39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45847 } }
I20260925 22:00:03.902915 23029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "935424c96fee48fbabc5526c8825bd39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935424c96fee48fbabc5526c8825bd39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45847 } } }
I20260925 22:00:03.902925 23030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 935424c96fee48fbabc5526c8825bd39. Latest consensus state: current_term: 1 leader_uuid: "935424c96fee48fbabc5526c8825bd39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935424c96fee48fbabc5526c8825bd39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45847 } } }
I20260925 22:00:03.903015 23029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:03.903021 23030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:03.903220 23038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:03.903729 23038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:03.903930 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 22957
I20260925 22:00:03.904064 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:03.904255 2331 external_mini_cluster.cc:1428] Setting key b3b5e0e0bf8a5b41947ced6516cce67d
I20260925 22:00:03.905850 23038 catalog_manager.cc:1347] Generated new cluster ID: 3804e72fd1784c49930bf80a31116070
I20260925 22:00:03.905905 23038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:03.910552 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:03.904728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39889 (local address 127.0.0.1:45847)
0925 22:00:03.904901 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:03.904911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:03.904945 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:00:03.905086 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:03.905093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:03.905160 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:03.905303 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:03.905691 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:03.907170 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:03.908328 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:03.908968 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:03.909115 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:03.909793 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:03.909797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:03.909797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:03.910035 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:03.910077 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:03.910094 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:03.910166 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:03.910231 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 22:00:03.910309 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260925 22:00:03.911559 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:03.916882 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:03.913132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45189 (local address 127.0.0.1:45847)
0925 22:00:03.913171 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:03.913173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:03.913320 (+ 147us) server_negotiation.cc:407] Connection header received
0925 22:00:03.913341 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:03.913343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:03.913363 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:03.913401 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:03.913570 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:03.914327 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:03.915392 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:03.915703 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:03.915757 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:03.916528 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:03.916529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:03.916530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:03.916535 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:03.916551 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:03.916556 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:03.916595 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:03.916764 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 22:00:03.916790 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 22:00:03.929419 23038 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:03.929976 23038 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:03.935536 23038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 935424c96fee48fbabc5526c8825bd39: Generated new TSK 0
I20260925 22:00:03.935742 23038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:03.942803 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 22957
2026-09-25T22:00:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:04 dist-test-slave-2g0f krb5kdc[23056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:04 dist-test-slave-2g0f krb5kdc[23056](info): set up 2 sockets
Sep 25 22:00:04 dist-test-slave-2g0f krb5kdc[23056](info): commencing operation
krb5kdc: starting...
W20260925 22:00:04.895677 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:04 dist-test-slave-2g0f krb5kdc[23056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:00:05.018708 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:05.100490 23069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:05.100695 23069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:05.100718 23069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:05.101931 23069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:05.101985 23069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:05.101999 23069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:05.102010 23069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:05.102020 23069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:05.103554 23069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:05.103803 23069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:05.104051 23069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:05.108695 23074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:05.108716 23077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:05.108705 23075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:05.108740 23069 server_base.cc:1034] running on GCE node
I20260925 22:00:05.109156 23069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:05.109442 23069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:05.110628 23069 hybrid_clock.cc:648] HybridClock initialized: now 1790373605110615 us; error 26 us; skew 500 ppm
Sep 25 22:00:05 dist-test-slave-2g0f krb5kdc[23056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:05.113237 23069 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:00:05.113714 23069 webserver.cc:460] Webserver started at http://127.0.0.1:35473/ using document root <none> and password file <none>
I20260925 22:00:05.113940 23069 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:05.113989 23069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:05.114085 23069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:05.115116 23069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "431537ce8fe94ac0a28103ea9c09f6f2"
format_stamp: "Formatted at 2026-09-25 22:00:05 on dist-test-slave-2g0f"
server_key: "0a1de0b15d431ba5a55fd103b2538bf3"
server_key_iv: "22058cd10e92adee4516c23d1615210f"
server_key_version: "encryptionkey@0"
I20260925 22:00:05.115437 23069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "431537ce8fe94ac0a28103ea9c09f6f2"
format_stamp: "Formatted at 2026-09-25 22:00:05 on dist-test-slave-2g0f"
server_key: "0a1de0b15d431ba5a55fd103b2538bf3"
server_key_iv: "22058cd10e92adee4516c23d1615210f"
server_key_version: "encryptionkey@0"
I20260925 22:00:05.116735 23069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 22:00:05.117475 23084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:05.117650 23069 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 22:00:05.117738 23069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "431537ce8fe94ac0a28103ea9c09f6f2"
format_stamp: "Formatted at 2026-09-25 22:00:05 on dist-test-slave-2g0f"
server_key: "0a1de0b15d431ba5a55fd103b2538bf3"
server_key_iv: "22058cd10e92adee4516c23d1615210f"
server_key_version: "encryptionkey@0"
I20260925 22:00:05.117800 23069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:05.121209 23069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:05.123725 23069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:05.123903 23069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:05.128672 23069 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34519
I20260925 22:00:05.128726 23136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34519 every 8 connection(s)
I20260925 22:00:05.129086 23069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:05.130118 23137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:05.132463 23137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2: Bootstrap starting.
I20260925 22:00:05.133013 23137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:05.133271 23137 log.cc:824] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:05.133575 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 23069
I20260925 22:00:05.133685 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:05.133867 2331 external_mini_cluster.cc:1428] Setting key 2037ca9b7769318f8f75fb299879a1d9
I20260925 22:00:05.133935 23137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2: No bootstrap required, opened a new log
I20260925 22:00:05.136044 23137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431537ce8fe94ac0a28103ea9c09f6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260925 22:00:05.136183 23137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:05.136204 23137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431537ce8fe94ac0a28103ea9c09f6f2, State: Initialized, Role: FOLLOWER
I20260925 22:00:05.136329 23137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431537ce8fe94ac0a28103ea9c09f6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260925 22:00:05.136404 23137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:05.136426 23137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:05.136488 23137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:05.136736 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:05.134358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59367 (local address 127.0.0.1:34519)
0925 22:00:05.134530 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:05.134539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:05.134566 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:00:05.135185 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:05.135193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:05.135220 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:05.135344 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:05.136069 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:05.136073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:05.136073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:05.136077 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:05.136114 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:05.136124 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:05.136226 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:05.136350 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 22:00:05.136391 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260925 22:00:05.137096 23137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431537ce8fe94ac0a28103ea9c09f6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260925 22:00:05.137207 23137 leader_election.cc:302] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 431537ce8fe94ac0a28103ea9c09f6f2; no voters:
I20260925 22:00:05.137367 23137 leader_election.cc:288] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:05.137539 23143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:05.137619 23137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:05.137652 23143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [term 1 LEADER]: Becoming Leader. State: Replica: 431537ce8fe94ac0a28103ea9c09f6f2, State: Running, Role: LEADER
I20260925 22:00:05.137669 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:05.137722 23143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431537ce8fe94ac0a28103ea9c09f6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260925 22:00:05.138023 23143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "431537ce8fe94ac0a28103ea9c09f6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431537ce8fe94ac0a28103ea9c09f6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } } }
I20260925 22:00:05.138090 23143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:05.138195 23143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 431537ce8fe94ac0a28103ea9c09f6f2. Latest consensus state: current_term: 1 leader_uuid: "431537ce8fe94ac0a28103ea9c09f6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431537ce8fe94ac0a28103ea9c09f6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } } }
I20260925 22:00:05.138234 23143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:05.138341 23149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:05.139000 23149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:05.140810 23149 catalog_manager.cc:1347] Generated new cluster ID: c4164935c01044c5bd1091462abb7855
I20260925 22:00:05.140861 23149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:05.141183 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:05.139051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56531 (local address 127.0.0.1:34519)
0925 22:00:05.139441 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:05.139443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:05.140168 (+ 725us) server_negotiation.cc:407] Connection header received
0925 22:00:05.140191 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:05.140192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:05.140197 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:05.140251 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:05.140917 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:05.140919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:05.140919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:05.140924 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:05.140938 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:05.140942 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:05.140981 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:05.141115 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 22:00:05.141125 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260925 22:00:05.156764 23149 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:05.157006 23149 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:05.159714 23149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 431537ce8fe94ac0a28103ea9c09f6f2: Generated new TSK 0
I20260925 22:00:05.159895 23149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:05.167236 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 23069
2026-09-25T22:00:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:05 dist-test-slave-2g0f krb5kdc[23168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:05 dist-test-slave-2g0f krb5kdc[23168](info): set up 2 sockets
Sep 25 22:00:05 dist-test-slave-2g0f krb5kdc[23168](info): commencing operation
krb5kdc: starting...
W20260925 22:00:06.128114 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:06 dist-test-slave-2g0f krb5kdc[23168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:00:06.244763 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:06.327116 23181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:06.327325 23181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:06.327356 23181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:06.328506 23181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:06.328553 23181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:06.328572 23181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:06.328589 23181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:06.328605 23181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:06.330156 23181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:06.330399 23181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:06.330603 23181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:06.335246 23187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:06.335273 23189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:06.335386 23181 server_base.cc:1034] running on GCE node
W20260925 22:00:06.335274 23186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:06.335749 23181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:06.336066 23181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:06.337242 23181 hybrid_clock.cc:648] HybridClock initialized: now 1790373606337230 us; error 26 us; skew 500 ppm
Sep 25 22:00:06 dist-test-slave-2g0f krb5kdc[23168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:06.339661 23181 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:00:06.340083 23181 webserver.cc:460] Webserver started at http://127.0.0.1:45893/ using document root <none> and password file <none>
I20260925 22:00:06.340291 23181 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:06.340330 23181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:06.340452 23181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:06.341470 23181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "ae8539420601499394b59c529f726cbc"
format_stamp: "Formatted at 2026-09-25 22:00:06 on dist-test-slave-2g0f"
server_key: "0a9c379f976e4b2167f203ebcdeafdea"
server_key_iv: "f8eee73f0e50969b07c482ff5d5337ca"
server_key_version: "encryptionkey@0"
I20260925 22:00:06.341768 23181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "ae8539420601499394b59c529f726cbc"
format_stamp: "Formatted at 2026-09-25 22:00:06 on dist-test-slave-2g0f"
server_key: "0a9c379f976e4b2167f203ebcdeafdea"
server_key_iv: "f8eee73f0e50969b07c482ff5d5337ca"
server_key_version: "encryptionkey@0"
I20260925 22:00:06.343037 23181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:00:06.343745 23196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:06.343906 23181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:00:06.343987 23181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "ae8539420601499394b59c529f726cbc"
format_stamp: "Formatted at 2026-09-25 22:00:06 on dist-test-slave-2g0f"
server_key: "0a9c379f976e4b2167f203ebcdeafdea"
server_key_iv: "f8eee73f0e50969b07c482ff5d5337ca"
server_key_version: "encryptionkey@0"
I20260925 22:00:06.344046 23181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:06.357220 23181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:06.359511 23181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:06.359656 23181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:06.364068 23181 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36545
I20260925 22:00:06.364209 23248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36545 every 8 connection(s)
I20260925 22:00:06.364639 23181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:06.365710 23249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:06.368422 23249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc: Bootstrap starting.
I20260925 22:00:06.369046 23249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:06.369280 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 23181
I20260925 22:00:06.369333 23249 log.cc:824] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:06.369383 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:06.369573 2331 external_mini_cluster.cc:1428] Setting key 20b61db5bd44610b4dd829c1e7c0d7c0
I20260925 22:00:06.370134 23249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc: No bootstrap required, opened a new log
I20260925 22:00:06.371991 23249 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8539420601499394b59c529f726cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260925 22:00:06.372141 23249 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:06.372164 23249 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8539420601499394b59c529f726cbc, State: Initialized, Role: FOLLOWER
I20260925 22:00:06.372279 23249 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8539420601499394b59c529f726cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260925 22:00:06.372356 23249 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:06.372381 23249 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:06.372430 23249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:06.373312 23249 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8539420601499394b59c529f726cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260925 22:00:06.373407 23249 leader_election.cc:302] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae8539420601499394b59c529f726cbc; no voters:
I20260925 22:00:06.373564 23249 leader_election.cc:288] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:06.373634 23254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:06.373824 23249 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:06.373852 23254 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [term 1 LEADER]: Becoming Leader. State: Replica: ae8539420601499394b59c529f726cbc, State: Running, Role: LEADER
I20260925 22:00:06.374006 23254 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8539420601499394b59c529f726cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260925 22:00:06.374359 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae8539420601499394b59c529f726cbc. Latest consensus state: current_term: 1 leader_uuid: "ae8539420601499394b59c529f726cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8539420601499394b59c529f726cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } } }
I20260925 22:00:06.374471 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:06.374856 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae8539420601499394b59c529f726cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8539420601499394b59c529f726cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } } }
I20260925 22:00:06.374933 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:06.375054 23262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:06.375535 23262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:06.377342 23262 catalog_manager.cc:1347] Generated new cluster ID: d7e8289603d74b6ca26800f0e45d978a
I20260925 22:00:06.377396 23262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 22:00:06 dist-test-slave-2g0f krb5kdc[23168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:00:06.383487 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:06.370123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:54847 (local address 127.0.0.1:36545)
0925 22:00:06.370302 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:06.370312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:06.370339 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:00:06.371970 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:06.371979 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:06.372050 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:06.372953 (+ 903us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:06.373052 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:06.374499 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:06.375618 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:06.376256 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:06.376733 (+ 477us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:06.379329 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:06.379333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:06.379333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:06.379347 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:06.381498 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:06.382048 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:06.382051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:06.382051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:06.382197 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:06.382510 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:06.382511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:06.382511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:06.382819 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:06.382970 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:06.383131 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 22:00:06.383256 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260925 22:00:06.384917 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:06.394078 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:06.386575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:44493 (local address 127.0.0.1:36545)
0925 22:00:06.386609 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:06.386611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:06.386618 (+ 7us) server_negotiation.cc:407] Connection header received
0925 22:00:06.386673 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:06.386674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:06.386695 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:06.386746 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:06.387380 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:06.388154 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:06.389205 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:06.389542 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:06.389595 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:06.391080 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:06.391081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:06.391081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:06.391089 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:06.392688 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:06.393151 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:06.393152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:06.393152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:06.393210 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:06.393413 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:06.393413 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:06.393413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:06.393539 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:06.393639 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:06.393789 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 22:00:06.393870 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 22:00:06.399765 23262 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:06.400336 23262 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:06.408684 23262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae8539420601499394b59c529f726cbc: Generated new TSK 0
I20260925 22:00:06.408932 23262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:06.421553 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 23181
W20260925 22:00:06.425365 23166 connection.cc:535] client connection to 127.0.0.1:36545 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36545): Connection reset by peer (error 104)
2026-09-25T22:00:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:06 dist-test-slave-2g0f krb5kdc[23280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:06 dist-test-slave-2g0f krb5kdc[23280](info): set up 2 sockets
Sep 25 22:00:06 dist-test-slave-2g0f krb5kdc[23280](info): commencing operation
krb5kdc: starting...
W20260925 22:00:07.371739 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:07 dist-test-slave-2g0f krb5kdc[23280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:00:07.488685 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:07.568149 23293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:07.568373 23293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:07.568406 23293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:07.569625 23293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:07.569669 23293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:07.569689 23293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:07.569705 23293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:07.569721 23293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:07.571256 23293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:07.571512 23293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:07.571727 23293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:07.576640 23299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:07.576671 23298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:07.576671 23301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:07.577029 23293 server_base.cc:1034] running on GCE node
I20260925 22:00:07.577239 23293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:07.577497 23293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:07.578649 23293 hybrid_clock.cc:648] HybridClock initialized: now 1790373607578626 us; error 39 us; skew 500 ppm
Sep 25 22:00:07 dist-test-slave-2g0f krb5kdc[23280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:07.581414 23293 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:00:07.581846 23293 webserver.cc:460] Webserver started at http://127.0.0.1:37009/ using document root <none> and password file <none>
I20260925 22:00:07.582062 23293 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:07.582120 23293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:07.582247 23293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:07.583362 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "2a87a8da910d4c12b6e18aa5bba54ad2"
format_stamp: "Formatted at 2026-09-25 22:00:07 on dist-test-slave-2g0f"
server_key: "50db1221e54fbbac70b9e72cbe80bf9b"
server_key_iv: "47eafa13b0ef9f7239464e31c86963b8"
server_key_version: "encryptionkey@0"
I20260925 22:00:07.583710 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "2a87a8da910d4c12b6e18aa5bba54ad2"
format_stamp: "Formatted at 2026-09-25 22:00:07 on dist-test-slave-2g0f"
server_key: "50db1221e54fbbac70b9e72cbe80bf9b"
server_key_iv: "47eafa13b0ef9f7239464e31c86963b8"
server_key_version: "encryptionkey@0"
I20260925 22:00:07.585098 23293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:00:07.585903 23308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:07.586241 23293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 22:00:07.586346 23293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "2a87a8da910d4c12b6e18aa5bba54ad2"
format_stamp: "Formatted at 2026-09-25 22:00:07 on dist-test-slave-2g0f"
server_key: "50db1221e54fbbac70b9e72cbe80bf9b"
server_key_iv: "47eafa13b0ef9f7239464e31c86963b8"
server_key_version: "encryptionkey@0"
I20260925 22:00:07.586419 23293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:07.595901 23293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:07.598117 23293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:07.598259 23293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:07.602099 23293 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43907
I20260925 22:00:07.602139 23360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43907 every 8 connection(s)
I20260925 22:00:07.602471 23293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:07.603485 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 23293
I20260925 22:00:07.603552 23361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:07.603605 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:07.603802 2331 external_mini_cluster.cc:1428] Setting key 7af1380bcf6591865a93cd0694aa95b1
I20260925 22:00:07.606938 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2: Bootstrap starting.
I20260925 22:00:07.607654 23361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:07.608131 23361 log.cc:824] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:07.609037 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2: No bootstrap required, opened a new log
I20260925 22:00:07.609534 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:07.604385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:44507 (local address 127.0.0.1:43907)
0925 22:00:07.604573 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:07.604581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:07.604607 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:00:07.605190 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:07.605198 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:07.605252 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:07.605371 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:07.605621 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:07.606510 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:07.607543 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:07.608039 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:07.608134 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:07.608909 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:07.608912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:07.608913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:07.609075 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:07.609116 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:07.609133 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:07.609212 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:07.609332 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 22:00:07.609361 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260925 22:00:07.610764 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:07.610844 23361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a87a8da910d4c12b6e18aa5bba54ad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43907 } }
I20260925 22:00:07.611023 23361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:07.611060 23361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a87a8da910d4c12b6e18aa5bba54ad2, State: Initialized, Role: FOLLOWER
I20260925 22:00:07.611169 23361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a87a8da910d4c12b6e18aa5bba54ad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43907 } }
I20260925 22:00:07.611239 23361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:07.611279 23361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:07.611325 23361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:07.612146 23361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a87a8da910d4c12b6e18aa5bba54ad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43907 } }
I20260925 22:00:07.612255 23361 leader_election.cc:302] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a87a8da910d4c12b6e18aa5bba54ad2; no voters:
I20260925 22:00:07.612433 23361 leader_election.cc:288] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:07.612510 23371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:07.612666 23371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [term 1 LEADER]: Becoming Leader. State: Replica: 2a87a8da910d4c12b6e18aa5bba54ad2, State: Running, Role: LEADER
I20260925 22:00:07.612696 23361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:07.612809 23371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a87a8da910d4c12b6e18aa5bba54ad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43907 } }
I20260925 22:00:07.613314 23373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a87a8da910d4c12b6e18aa5bba54ad2. Latest consensus state: current_term: 1 leader_uuid: "2a87a8da910d4c12b6e18aa5bba54ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a87a8da910d4c12b6e18aa5bba54ad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43907 } } }
I20260925 22:00:07.613405 23373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:07.613552 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a87a8da910d4c12b6e18aa5bba54ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a87a8da910d4c12b6e18aa5bba54ad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43907 } } }
I20260925 22:00:07.613602 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:07.613674 23377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:07.614209 23377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:07.616027 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:07.612437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:34301 (local address 127.0.0.1:43907)
0925 22:00:07.612475 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:07.612477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:07.612490 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:00:07.612636 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:07.612637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:07.612648 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:07.612675 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:07.612940 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:07.613649 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:07.614672 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:07.614943 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:07.614989 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:07.615738 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:07.615739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:07.615740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:07.615744 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:07.615759 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:07.615763 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:07.615814 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:07.615943 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 22:00:07.615956 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 22:00:07.616487 23377 catalog_manager.cc:1347] Generated new cluster ID: 103129afb911400f97ae2d9a60c969fe
I20260925 22:00:07.616537 23377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:07.620603 23377 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:07.621318 23377 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:07.634028 23377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a87a8da910d4c12b6e18aa5bba54ad2: Generated new TSK 0
I20260925 22:00:07.634305 23377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:07.643258 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 23293
W20260925 22:00:07.647033 23278 connection.cc:535] client connection to 127.0.0.1:43907 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43907): Connection reset by peer (error 104)
2026-09-25T22:00:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:07 dist-test-slave-2g0f krb5kdc[23392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:07 dist-test-slave-2g0f krb5kdc[23392](info): set up 2 sockets
Sep 25 22:00:07 dist-test-slave-2g0f krb5kdc[23392](info): commencing operation
krb5kdc: starting...
W20260925 22:00:08.603763 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:08 dist-test-slave-2g0f krb5kdc[23392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:00:08.722165 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:08.804143 23405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:08.804371 23405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:08.804396 23405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:08.805615 23405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:08.805660 23405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:08.805671 23405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:08.805683 23405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:08.805694 23405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:08.807238 23405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:08.807456 23405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:08.807641 23405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:08.812525 23410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:08.812529 23413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:08.812525 23411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:08.812757 23405 server_base.cc:1034] running on GCE node
I20260925 22:00:08.813117 23405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:08.813403 23405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:08.814566 23405 hybrid_clock.cc:648] HybridClock initialized: now 1790373608814542 us; error 37 us; skew 500 ppm
Sep 25 22:00:08 dist-test-slave-2g0f krb5kdc[23392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:08.817288 23405 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:00:08.817735 23405 webserver.cc:460] Webserver started at http://127.0.0.1:38273/ using document root <none> and password file <none>
I20260925 22:00:08.817960 23405 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:08.818011 23405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:08.818148 23405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:08.819219 23405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "671691973df64f8485b66df207924c56"
format_stamp: "Formatted at 2026-09-25 22:00:08 on dist-test-slave-2g0f"
server_key: "5c825df64c88c9815a6f47e750c22358"
server_key_iv: "74f7c48b7a1b1faf1d43f37841f71d7b"
server_key_version: "encryptionkey@0"
I20260925 22:00:08.819551 23405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "671691973df64f8485b66df207924c56"
format_stamp: "Formatted at 2026-09-25 22:00:08 on dist-test-slave-2g0f"
server_key: "5c825df64c88c9815a6f47e750c22358"
server_key_iv: "74f7c48b7a1b1faf1d43f37841f71d7b"
server_key_version: "encryptionkey@0"
I20260925 22:00:08.820991 23405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 22:00:08.821717 23420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:08.821866 23405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 22:00:08.821995 23405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "671691973df64f8485b66df207924c56"
format_stamp: "Formatted at 2026-09-25 22:00:08 on dist-test-slave-2g0f"
server_key: "5c825df64c88c9815a6f47e750c22358"
server_key_iv: "74f7c48b7a1b1faf1d43f37841f71d7b"
server_key_version: "encryptionkey@0"
I20260925 22:00:08.822062 23405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:08.825357 23405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:08.827415 23405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:08.827543 23405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:08.831144 23405 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33955
I20260925 22:00:08.831193 23472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33955 every 8 connection(s)
I20260925 22:00:08.831493 23405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:08.832436 23473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:08.834595 23473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56: Bootstrap starting.
I20260925 22:00:08.835111 23473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:08.835351 23473 log.cc:824] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:08.836015 23473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56: No bootstrap required, opened a new log
I20260925 22:00:08.836825 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 23405
I20260925 22:00:08.836951 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:08.837136 2331 external_mini_cluster.cc:1428] Setting key 76a877dc66a2e3ab70456dcd7ae80972
I20260925 22:00:08.837771 23473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "671691973df64f8485b66df207924c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33955 } }
I20260925 22:00:08.837940 23473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:08.837982 23473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671691973df64f8485b66df207924c56, State: Initialized, Role: FOLLOWER
I20260925 22:00:08.838099 23473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "671691973df64f8485b66df207924c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33955 } }
I20260925 22:00:08.838176 23473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:08.838250 23473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:08.838302 23473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:08.839186 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:08.837643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:53647 (local address 127.0.0.1:33955)
0925 22:00:08.837828 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:08.837837 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:08.837868 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:00:08.837985 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:08.837993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:08.838024 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:08.838165 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:08.838751 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:08.838755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:08.838755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:08.838759 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:08.838804 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:08.838820 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:08.838893 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:08.838927 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 22:00:08.838981 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260925 22:00:08.839532 23473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "671691973df64f8485b66df207924c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33955 } }
I20260925 22:00:08.839633 23473 leader_election.cc:302] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 671691973df64f8485b66df207924c56; no voters:
I20260925 22:00:08.839781 23473 leader_election.cc:288] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:08.839802 23478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:08.839967 23478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [term 1 LEADER]: Becoming Leader. State: Replica: 671691973df64f8485b66df207924c56, State: Running, Role: LEADER
I20260925 22:00:08.840037 23473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:08.840094 23478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "671691973df64f8485b66df207924c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33955 } }
I20260925 22:00:08.840171 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:08.840498 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 671691973df64f8485b66df207924c56. Latest consensus state: current_term: 1 leader_uuid: "671691973df64f8485b66df207924c56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "671691973df64f8485b66df207924c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33955 } } }
I20260925 22:00:08.840914 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:08.840816 23479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "671691973df64f8485b66df207924c56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "671691973df64f8485b66df207924c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33955 } } }
I20260925 22:00:08.841316 23479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:08.841339 23490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:08.841761 23490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:08.844446 23490 catalog_manager.cc:1347] Generated new cluster ID: 0d99daab0d03493da403735c79e709c6
I20260925 22:00:08.844501 23490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:08.844898 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:08.842783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:48777 (local address 127.0.0.1:33955)
0925 22:00:08.842816 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:08.842818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:08.844037 (+ 1219us) server_negotiation.cc:407] Connection header received
0925 22:00:08.844089 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:08.844090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:08.844096 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:08.844148 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:08.844716 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:08.844717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:08.844717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:08.844721 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:08.844734 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:08.844738 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:08.844783 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:08.844825 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 22:00:08.844838 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 22:00:08.870709 23490 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:08.870929 23490 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:08.874073 23490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 671691973df64f8485b66df207924c56: Generated new TSK 0
I20260925 22:00:08.874281 23490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:08.881502 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 23405
2026-09-25T22:00:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1238 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:08 dist-test-slave-2g0f krb5kdc[23504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:08 dist-test-slave-2g0f krb5kdc[23504](info): set up 2 sockets
Sep 25 22:00:08 dist-test-slave-2g0f krb5kdc[23504](info): commencing operation
krb5kdc: starting...
W20260925 22:00:09.843740 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:09 dist-test-slave-2g0f krb5kdc[23504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:00:09.963716 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:10.047300 23517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:10.047518 23517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:10.047539 23517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:10.048643 23517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:10.048676 23517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:10.048688 23517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:10.048699 23517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:10.048709 23517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:10.050208 23517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:10.050426 23517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:10.050604 23517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:10.055881 23522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:10.055964 23523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:10.055956 23525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:10.056250 23517 server_base.cc:1034] running on GCE node
I20260925 22:00:10.056527 23517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:10.056839 23517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:10.058004 23517 hybrid_clock.cc:648] HybridClock initialized: now 1790373610057986 us; error 34 us; skew 500 ppm
Sep 25 22:00:10 dist-test-slave-2g0f krb5kdc[23504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:10.060791 23517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:00:10.061336 23517 webserver.cc:460] Webserver started at http://127.0.0.1:32935/ using document root <none> and password file <none>
I20260925 22:00:10.061573 23517 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:10.061630 23517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:10.061757 23517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:10.062963 23517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "c5651ecc02c948549385d2e322e71d61"
format_stamp: "Formatted at 2026-09-25 22:00:10 on dist-test-slave-2g0f"
server_key: "5089909b30c18e0957ddb06bb0359003"
server_key_iv: "4a48dc8ffbb2073eab7862c7090ebfb4"
server_key_version: "encryptionkey@0"
I20260925 22:00:10.063482 23517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "c5651ecc02c948549385d2e322e71d61"
format_stamp: "Formatted at 2026-09-25 22:00:10 on dist-test-slave-2g0f"
server_key: "5089909b30c18e0957ddb06bb0359003"
server_key_iv: "4a48dc8ffbb2073eab7862c7090ebfb4"
server_key_version: "encryptionkey@0"
I20260925 22:00:10.064971 23517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 22:00:10.065824 23532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:10.066039 23517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:00:10.066135 23517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "c5651ecc02c948549385d2e322e71d61"
format_stamp: "Formatted at 2026-09-25 22:00:10 on dist-test-slave-2g0f"
server_key: "5089909b30c18e0957ddb06bb0359003"
server_key_iv: "4a48dc8ffbb2073eab7862c7090ebfb4"
server_key_version: "encryptionkey@0"
I20260925 22:00:10.066206 23517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:10.077437 23517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:10.080075 23517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:10.080238 23517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:10.084602 23517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37325
I20260925 22:00:10.084697 23584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37325 every 8 connection(s)
I20260925 22:00:10.085022 23517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:10.086076 23585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:10.089071 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 23517
I20260925 22:00:10.089186 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:10.089371 2331 external_mini_cluster.cc:1428] Setting key 7aa3bab11aeba4237df79a419a1fba29
I20260925 22:00:10.090265 23585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61: Bootstrap starting.
I20260925 22:00:10.091197 23585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:10.091501 23585 log.cc:824] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:10.092338 23585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61: No bootstrap required, opened a new log
I20260925 22:00:10.094408 23585 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5651ecc02c948549385d2e322e71d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260925 22:00:10.094579 23585 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:10.094614 23585 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5651ecc02c948549385d2e322e71d61, State: Initialized, Role: FOLLOWER
I20260925 22:00:10.094750 23585 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5651ecc02c948549385d2e322e71d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260925 22:00:10.094981 23585 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:10.095060 23585 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:10.095117 23585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:10.096030 23585 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5651ecc02c948549385d2e322e71d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260925 22:00:10.096190 23585 leader_election.cc:302] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5651ecc02c948549385d2e322e71d61; no voters:
I20260925 22:00:10.096391 23585 leader_election.cc:288] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:10.096454 23590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:10.096674 23590 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [term 1 LEADER]: Becoming Leader. State: Replica: c5651ecc02c948549385d2e322e71d61, State: Running, Role: LEADER
I20260925 22:00:10.096696 23585 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:10.096879 23590 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5651ecc02c948549385d2e322e71d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
Sep 25 22:00:10 dist-test-slave-2g0f krb5kdc[23504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:00:10.097481 23592 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5651ecc02c948549385d2e322e71d61. Latest consensus state: current_term: 1 leader_uuid: "c5651ecc02c948549385d2e322e71d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5651ecc02c948549385d2e322e71d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } } }
I20260925 22:00:10.097566 23592 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:10.097765 23591 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5651ecc02c948549385d2e322e71d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5651ecc02c948549385d2e322e71d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } } }
I20260925 22:00:10.097823 23591 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:10.097930 23599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:10.098688 23599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:10.100463 23599 catalog_manager.cc:1347] Generated new cluster ID: 1c9a20f9fe4f4d5b8748e9e179e23790
I20260925 22:00:10.100514 23599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:10.101823 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:10.090156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:39535 (local address 127.0.0.1:37325)
0925 22:00:10.090540 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:10.090550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:10.090583 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:00:10.091205 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:10.091214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:10.091300 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:10.091454 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:10.092228 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:10.093170 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:10.094056 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:10.094685 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:10.094967 (+ 282us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:10.098004 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:10.098008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:10.098009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:10.098024 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:10.099977 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:10.100490 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:10.100494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:10.100494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:10.100617 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:10.100867 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:10.100868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:10.100868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:10.101184 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:10.101365 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:10.101469 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 22:00:10.101599 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":178,"threads_started":1}
I20260925 22:00:10.102818 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:10.107705 23599 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:10.108385 23599 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:10.112026 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:10.104415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:52347 (local address 127.0.0.1:37325)
0925 22:00:10.104447 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:10.104448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:10.104456 (+ 8us) server_negotiation.cc:407] Connection header received
0925 22:00:10.104502 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:10.104502 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:10.104511 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:10.104562 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:10.105192 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:10.106240 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:10.107322 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:10.107593 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:10.107649 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:10.109154 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:10.109156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:10.109156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:10.109163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:10.110694 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:10.111128 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:10.111129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:10.111129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:10.111185 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:10.111387 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:10.111388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:10.111388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:10.111502 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:10.111597 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:10.111739 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 22:00:10.111798 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 22:00:10.114480 23599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5651ecc02c948549385d2e322e71d61: Generated new TSK 0
I20260925 22:00:10.114691 23599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:10.118254 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 23517
W20260925 22:00:10.122113 23502 connection.cc:535] client connection to 127.0.0.1:37325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37325): Connection reset by peer (error 104)
2026-09-25T22:00:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:10 dist-test-slave-2g0f krb5kdc[23616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:10 dist-test-slave-2g0f krb5kdc[23616](info): set up 2 sockets
Sep 25 22:00:10 dist-test-slave-2g0f krb5kdc[23616](info): commencing operation
krb5kdc: starting...
W20260925 22:00:11.071856 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:11 dist-test-slave-2g0f krb5kdc[23616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:00:11.198891 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:11.285313 23629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:11.285557 23629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:11.285590 23629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:11.286895 23629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:11.286943 23629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:11.286957 23629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:11.286968 23629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:11.286979 23629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:11.288614 23629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:11.288854 23629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:11.289042 23629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:00:11.294116 23629 server_base.cc:1034] running on GCE node
W20260925 22:00:11.294152 23634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:11.294157 23635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:11.294293 23637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:11.294538 23629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:11.294826 23629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:11.295953 23629 hybrid_clock.cc:648] HybridClock initialized: now 1790373611295934 us; error 36 us; skew 500 ppm
Sep 25 22:00:11 dist-test-slave-2g0f krb5kdc[23616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:11.298410 23629 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:00:11.298868 23629 webserver.cc:460] Webserver started at http://127.0.0.1:37191/ using document root <none> and password file <none>
I20260925 22:00:11.299125 23629 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:11.299173 23629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:11.299286 23629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:11.300406 23629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "1136a47f90ad4fb7bf307da1ab7cc9da"
format_stamp: "Formatted at 2026-09-25 22:00:11 on dist-test-slave-2g0f"
server_key: "b9d94be20dea8ecde0740ecb11f17205"
server_key_iv: "64497d50513a683bb997a9b5dbe03e17"
server_key_version: "encryptionkey@0"
I20260925 22:00:11.300726 23629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "1136a47f90ad4fb7bf307da1ab7cc9da"
format_stamp: "Formatted at 2026-09-25 22:00:11 on dist-test-slave-2g0f"
server_key: "b9d94be20dea8ecde0740ecb11f17205"
server_key_iv: "64497d50513a683bb997a9b5dbe03e17"
server_key_version: "encryptionkey@0"
I20260925 22:00:11.302042 23629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 22:00:11.302795 23644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:11.303011 23629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:00:11.303100 23629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "1136a47f90ad4fb7bf307da1ab7cc9da"
format_stamp: "Formatted at 2026-09-25 22:00:11 on dist-test-slave-2g0f"
server_key: "b9d94be20dea8ecde0740ecb11f17205"
server_key_iv: "64497d50513a683bb997a9b5dbe03e17"
server_key_version: "encryptionkey@0"
I20260925 22:00:11.303164 23629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:11.321992 23629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:11.323979 23629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:11.324126 23629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:11.328045 23629 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41153
I20260925 22:00:11.328084 23696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41153 every 8 connection(s)
I20260925 22:00:11.328408 23629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:11.329272 23697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:11.331508 23697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da: Bootstrap starting.
I20260925 22:00:11.332063 23697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:11.332302 23697 log.cc:824] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:11.332947 23697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da: No bootstrap required, opened a new log
I20260925 22:00:11.333946 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 23629
I20260925 22:00:11.334053 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:11.334242 2331 external_mini_cluster.cc:1428] Setting key 93f361c827c0a4e7ca5e24e13bdb582f
I20260925 22:00:11.335021 23697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1136a47f90ad4fb7bf307da1ab7cc9da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260925 22:00:11.335155 23697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:11.335184 23697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1136a47f90ad4fb7bf307da1ab7cc9da, State: Initialized, Role: FOLLOWER
I20260925 22:00:11.335325 23697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1136a47f90ad4fb7bf307da1ab7cc9da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260925 22:00:11.335393 23697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:11.335423 23697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:11.335469 23697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:11.336014 23697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1136a47f90ad4fb7bf307da1ab7cc9da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260925 22:00:11.336105 23697 leader_election.cc:302] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1136a47f90ad4fb7bf307da1ab7cc9da; no voters:
I20260925 22:00:11.336237 23697 leader_election.cc:288] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:11.336268 23702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:11.336416 23702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [term 1 LEADER]: Becoming Leader. State: Replica: 1136a47f90ad4fb7bf307da1ab7cc9da, State: Running, Role: LEADER
I20260925 22:00:11.336464 23697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:11.336562 23702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1136a47f90ad4fb7bf307da1ab7cc9da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260925 22:00:11.336855 23702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1136a47f90ad4fb7bf307da1ab7cc9da. Latest consensus state: current_term: 1 leader_uuid: "1136a47f90ad4fb7bf307da1ab7cc9da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1136a47f90ad4fb7bf307da1ab7cc9da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } } }
I20260925 22:00:11.336922 23702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:11.337124 23704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1136a47f90ad4fb7bf307da1ab7cc9da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1136a47f90ad4fb7bf307da1ab7cc9da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } } }
I20260925 22:00:11.337195 23704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:11.337302 23708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:11.337776 23708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:11.339310 23708 catalog_manager.cc:1347] Generated new cluster ID: c14808811ccd4542a814e21565290f6a
I20260925 22:00:11.339360 23708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:11.340169 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:11.334750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:59299 (local address 127.0.0.1:41153)
0925 22:00:11.334924 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:11.334934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:11.334965 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:00:11.335062 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:11.335067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:11.335113 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:11.335229 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:11.335432 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:11.337095 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:11.338180 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:11.338761 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:11.338865 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:11.339580 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:11.339584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:11.339585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:11.339741 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:11.339771 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:11.339781 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:11.339874 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:11.339950 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 22:00:11.339977 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260925 22:00:11.341015 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:11.344904 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:11.342188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:59259 (local address 127.0.0.1:41153)
0925 22:00:11.342224 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:11.342225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:11.342338 (+ 113us) server_negotiation.cc:407] Connection header received
0925 22:00:11.342407 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:11.342407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:11.342417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:11.342460 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:11.342575 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:11.343026 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:11.343682 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:11.344071 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:11.344128 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:11.344679 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:11.344681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:11.344681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:11.344685 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:11.344697 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:11.344702 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:11.344749 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:11.344814 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 22:00:11.344825 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 22:00:11.360756 23708 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:11.361340 23708 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:11.365366 23708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1136a47f90ad4fb7bf307da1ab7cc9da: Generated new TSK 0
I20260925 22:00:11.365567 23708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:11.376336 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 23629
W20260925 22:00:11.380390 23614 connection.cc:535] client connection to 127.0.0.1:41153 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41153): Connection reset by peer (error 104)
2026-09-25T22:00:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1258 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:11 dist-test-slave-2g0f krb5kdc[23728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:11 dist-test-slave-2g0f krb5kdc[23728](info): set up 2 sockets
Sep 25 22:00:11 dist-test-slave-2g0f krb5kdc[23728](info): commencing operation
krb5kdc: starting...
W20260925 22:00:12.331763 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:12 dist-test-slave-2g0f krb5kdc[23728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:00:12.450723 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:12.533341 23741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:12.533545 23741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:12.533566 23741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:12.534643 23741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:12.534677 23741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:12.534689 23741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:12.534700 23741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:12.534710 23741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:12.536183 23741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:12.536372 23741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:12.536540 23741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:12.541014 23749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:12.541128 23746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:12.541149 23747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:12.541332 23741 server_base.cc:1034] running on GCE node
I20260925 22:00:12.541550 23741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:12.541793 23741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:12.542928 23741 hybrid_clock.cc:648] HybridClock initialized: now 1790373612542918 us; error 25 us; skew 500 ppm
Sep 25 22:00:12 dist-test-slave-2g0f krb5kdc[23728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:12.550392 23741 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:00:12.550849 23741 webserver.cc:460] Webserver started at http://127.0.0.1:38851/ using document root <none> and password file <none>
I20260925 22:00:12.551038 23741 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:12.551077 23741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:12.551172 23741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:12.552261 23741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "7763924148374a82a7c555595e62013a"
format_stamp: "Formatted at 2026-09-25 22:00:12 on dist-test-slave-2g0f"
server_key: "a0c0ca3ab8f4ac05509891754356fd98"
server_key_iv: "c9c9b5806e87f74e308f7a1cfb320ebe"
server_key_version: "encryptionkey@0"
I20260925 22:00:12.552590 23741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "7763924148374a82a7c555595e62013a"
format_stamp: "Formatted at 2026-09-25 22:00:12 on dist-test-slave-2g0f"
server_key: "a0c0ca3ab8f4ac05509891754356fd98"
server_key_iv: "c9c9b5806e87f74e308f7a1cfb320ebe"
server_key_version: "encryptionkey@0"
I20260925 22:00:12.553839 23741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 22:00:12.554531 23756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:12.554777 23741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:00:12.554872 23741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "7763924148374a82a7c555595e62013a"
format_stamp: "Formatted at 2026-09-25 22:00:12 on dist-test-slave-2g0f"
server_key: "a0c0ca3ab8f4ac05509891754356fd98"
server_key_iv: "c9c9b5806e87f74e308f7a1cfb320ebe"
server_key_version: "encryptionkey@0"
I20260925 22:00:12.554955 23741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:12.558387 23741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:12.560626 23741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:12.560750 23741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:12.564723 23741 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40235
I20260925 22:00:12.564761 23808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40235 every 8 connection(s)
I20260925 22:00:12.565088 23741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:12.565325 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 23741
I20260925 22:00:12.565454 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:12.565694 2331 external_mini_cluster.cc:1428] Setting key 8aeae01092de862f7ab2bb5f697cd7b2
I20260925 22:00:12.566325 23809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:12.568517 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:12.566359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:57857 (local address 127.0.0.1:40235)
0925 22:00:12.566563 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:12.566574 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:12.566602 (+ 28us) server_negotiation.cc:407] Connection header received
0925 22:00:12.567107 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:12.567114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:12.567142 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:12.567263 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:12.568068 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:12.568072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:12.568073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:12.568077 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:12.568120 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:12.568132 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:12.568196 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:12.568276 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 22:00:12.568314 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":110,"threads_started":1}
I20260925 22:00:12.569428 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:12.569857 23809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a: Bootstrap starting.
I20260925 22:00:12.570647 23809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:12.570978 23809 log.cc:824] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:12.571761 23809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a: No bootstrap required, opened a new log
I20260925 22:00:12.572156 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:12.571008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:57507 (local address 127.0.0.1:40235)
0925 22:00:12.571052 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:12.571054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:12.571205 (+ 151us) server_negotiation.cc:407] Connection header received
0925 22:00:12.571255 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:12.571256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:12.571261 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:12.571313 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:12.571941 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:12.571943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:12.571943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:12.571947 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:12.571964 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:12.571968 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:12.572023 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:12.572075 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 22:00:12.572089 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 22:00:12.573819 23809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7763924148374a82a7c555595e62013a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } }
I20260925 22:00:12.573975 23809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:12.574002 23809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7763924148374a82a7c555595e62013a, State: Initialized, Role: FOLLOWER
I20260925 22:00:12.574115 23809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7763924148374a82a7c555595e62013a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } }
I20260925 22:00:12.574200 23809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:12.574234 23809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:12.574282 23809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:12.574915 23809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7763924148374a82a7c555595e62013a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } }
I20260925 22:00:12.575011 23809 leader_election.cc:302] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7763924148374a82a7c555595e62013a; no voters:
I20260925 22:00:12.575207 23809 leader_election.cc:288] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:12.575415 23819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:12.575470 23809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:12.575712 23819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [term 1 LEADER]: Becoming Leader. State: Replica: 7763924148374a82a7c555595e62013a, State: Running, Role: LEADER
I20260925 22:00:12.575973 23819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7763924148374a82a7c555595e62013a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } }
I20260925 22:00:12.576664 23821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7763924148374a82a7c555595e62013a. Latest consensus state: current_term: 1 leader_uuid: "7763924148374a82a7c555595e62013a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7763924148374a82a7c555595e62013a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } } }
I20260925 22:00:12.576967 23821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:12.577279 23828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:12.577461 23820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7763924148374a82a7c555595e62013a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7763924148374a82a7c555595e62013a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } } }
I20260925 22:00:12.577561 23820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:12.577865 23828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:12.579560 23828 catalog_manager.cc:1347] Generated new cluster ID: 34f07bf2f95743e69a43d75a76c5d06a
I20260925 22:00:12.579610 23828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:12.614095 23828 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:12.614300 23828 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:12.622479 23828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7763924148374a82a7c555595e62013a: Generated new TSK 0
I20260925 22:00:12.622648 23828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:12.632269 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 23741
2026-09-25T22:00:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1255 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:12 dist-test-slave-2g0f krb5kdc[23840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:12 dist-test-slave-2g0f krb5kdc[23840](info): set up 2 sockets
Sep 25 22:00:12 dist-test-slave-2g0f krb5kdc[23840](info): commencing operation
krb5kdc: starting...
W20260925 22:00:13.592032 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:13 dist-test-slave-2g0f krb5kdc[23840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 22:00:13.711794 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:43661
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:43661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:13.794175 23853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:13.794411 23853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:13.794446 23853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:13.795702 23853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:13.795751 23853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:13.795770 23853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:13.795786 23853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:13.795801 23853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:13.797462 23853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:43661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:43661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:13.797735 23853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:13.797956 23853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:13.803076 23859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:13.803084 23861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:13.803119 23853 server_base.cc:1034] running on GCE node
W20260925 22:00:13.803088 23858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:13.803608 23853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:13.803995 23853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:13.805153 23853 hybrid_clock.cc:648] HybridClock initialized: now 1790373613805132 us; error 40 us; skew 500 ppm
Sep 25 22:00:13 dist-test-slave-2g0f krb5kdc[23840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373613, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:13.807770 23853 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 22:00:13.808283 23853 webserver.cc:460] Webserver started at http://127.2.70.254:43677/ using document root <none> and password file <none>
I20260925 22:00:13.808516 23853 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:13.808566 23853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:13.808699 23853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:13.809798 23853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "c2448c2b5e0d4f6d9c3cea97e891a9b6"
format_stamp: "Formatted at 2026-09-25 22:00:13 on dist-test-slave-2g0f"
server_key: "55082ac92def99eefd0d162e9e830936"
server_key_iv: "d2ee1d5c514e5fc317a32a876a1d7ba1"
server_key_version: "encryptionkey@0"
I20260925 22:00:13.810144 23853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "c2448c2b5e0d4f6d9c3cea97e891a9b6"
format_stamp: "Formatted at 2026-09-25 22:00:13 on dist-test-slave-2g0f"
server_key: "55082ac92def99eefd0d162e9e830936"
server_key_iv: "d2ee1d5c514e5fc317a32a876a1d7ba1"
server_key_version: "encryptionkey@0"
I20260925 22:00:13.811527 23853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 22:00:13.812322 23868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:13.812515 23853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:13.812608 23853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "c2448c2b5e0d4f6d9c3cea97e891a9b6"
format_stamp: "Formatted at 2026-09-25 22:00:13 on dist-test-slave-2g0f"
server_key: "55082ac92def99eefd0d162e9e830936"
server_key_iv: "d2ee1d5c514e5fc317a32a876a1d7ba1"
server_key_version: "encryptionkey@0"
I20260925 22:00:13.812673 23853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:13.829989 23853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:13.832131 23853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:13.832266 23853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:13.836097 23853 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:43661
I20260925 22:00:13.836143 23920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:43661 every 8 connection(s)
I20260925 22:00:13.836469 23853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:13.837217 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 23853
I20260925 22:00:13.837325 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:13.837415 23921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:13.837525 2331 external_mini_cluster.cc:1428] Setting key 7f2200e307c5b3c4d7273c04b4a9231c
I20260925 22:00:13.840435 23921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6: Bootstrap starting.
I20260925 22:00:13.840937 23921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:13.841198 23921 log.cc:824] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:13.841902 23921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6: No bootstrap required, opened a new log
I20260925 22:00:13.844178 23921 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2448c2b5e0d4f6d9c3cea97e891a9b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43661 } }
I20260925 22:00:13.844306 23921 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:13.844327 23921 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2448c2b5e0d4f6d9c3cea97e891a9b6, State: Initialized, Role: FOLLOWER
I20260925 22:00:13.844444 23921 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2448c2b5e0d4f6d9c3cea97e891a9b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43661 } }
I20260925 22:00:13.844512 23921 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:13.844533 23921 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 25 22:00:13 dist-test-slave-2g0f krb5kdc[23840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 22:00:13.844581 23921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:13.845083 23921 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2448c2b5e0d4f6d9c3cea97e891a9b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43661 } }
I20260925 22:00:13.845189 23921 leader_election.cc:302] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2448c2b5e0d4f6d9c3cea97e891a9b6; no voters:
I20260925 22:00:13.845336 23921 leader_election.cc:288] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:13.845381 23926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:13.845526 23926 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [term 1 LEADER]: Becoming Leader. State: Replica: c2448c2b5e0d4f6d9c3cea97e891a9b6, State: Running, Role: LEADER
I20260925 22:00:13.845587 23921 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:13.845614 23926 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2448c2b5e0d4f6d9c3cea97e891a9b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43661 } }
I20260925 22:00:13.845948 23928 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2448c2b5e0d4f6d9c3cea97e891a9b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2448c2b5e0d4f6d9c3cea97e891a9b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43661 } } }
I20260925 22:00:13.845965 23926 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2448c2b5e0d4f6d9c3cea97e891a9b6. Latest consensus state: current_term: 1 leader_uuid: "c2448c2b5e0d4f6d9c3cea97e891a9b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2448c2b5e0d4f6d9c3cea97e891a9b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43661 } } }
I20260925 22:00:13.846140 23926 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:13.846081 23928 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:13.846355 23935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:13.846855 23935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:13.848480 23935 catalog_manager.cc:1347] Generated new cluster ID: a0a99a9133434f1baed10f46a7efee85
I20260925 22:00:13.848528 23935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:13.848734 23924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:13.838051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60195 (local address 127.2.70.254:43661)
0925 22:00:13.838219 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:13.838243 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:13.838268 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:00:13.838790 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:13.838797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:13.838851 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:13.838957 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:13.839682 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:13.840946 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:13.841909 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:13.842386 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:13.842467 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:13.845065 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:13.845069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:13.845069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:13.845081 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:13.847099 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:13.847471 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:13.847474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:13.847475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:13.847559 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:13.847775 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:13.847775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:13.847776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:13.848110 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:13.848248 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:13.848477 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 22:00:13.848569 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260925 22:00:13.853677 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:13.856587 23935 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:13.857131 23935 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:13.863067 23935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2448c2b5e0d4f6d9c3cea97e891a9b6: Generated new TSK 0
I20260925 22:00:13.863302 23935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:13.869102 23924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:13.860231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35209 (local address 127.2.70.254:43661)
0925 22:00:13.860288 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:13.860290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:13.862950 (+ 2660us) server_negotiation.cc:407] Connection header received
0925 22:00:13.863063 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:13.863064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:13.863077 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:13.863130 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:13.863768 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:13.864347 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:13.865096 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:13.865340 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:13.865396 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:13.866611 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:13.866613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:13.866613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:13.866622 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:13.867996 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:13.868355 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:13.868357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:13.868357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:13.868411 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:13.868610 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:13.868611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:13.868611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:13.868722 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:13.868817 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:13.868936 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 22:00:13.869008 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 22:00:13.871155 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 23853
W20260925 22:00:13.875344 23838 connection.cc:535] client connection to 127.2.70.254:43661 recv error: Network error: failed to read from TLS socket (remote: 127.2.70.254:43661): Connection reset by peer (error 104)
2026-09-25T22:00:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1239 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:13 dist-test-slave-2g0f krb5kdc[23952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:13 dist-test-slave-2g0f krb5kdc[23952](info): set up 2 sockets
Sep 25 22:00:13 dist-test-slave-2g0f krb5kdc[23952](info): commencing operation
krb5kdc: starting...
W20260925 22:00:14.827675 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:14 dist-test-slave-2g0f krb5kdc[23952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 22:00:14.947000 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:43999
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:43999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:15.028273 23965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:15.028505 23965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:15.028538 23965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:15.029789 23965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:15.029841 23965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:15.029860 23965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:15.029877 23965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:15.029892 23965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:15.031715 23965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:43999
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:43999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:15.032069 23965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:15.032294 23965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:15.037490 23973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:15.037544 23970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:15.037644 23965 server_base.cc:1034] running on GCE node
W20260925 22:00:15.037506 23971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:15.037947 23965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:15.038230 23965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:15.039397 23965 hybrid_clock.cc:648] HybridClock initialized: now 1790373615039395 us; error 43 us; skew 500 ppm
Sep 25 22:00:15 dist-test-slave-2g0f krb5kdc[23952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373615, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:15.042165 23965 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 22:00:15.042658 23965 webserver.cc:460] Webserver started at http://127.2.70.254:37283/ using document root <none> and password file <none>
I20260925 22:00:15.042881 23965 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:15.042927 23965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:15.043023 23965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:15.044240 23965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "df69e898055146c284d2d61df4cbb54b"
format_stamp: "Formatted at 2026-09-25 22:00:15 on dist-test-slave-2g0f"
server_key: "9c21bb4a7be09f584117b1b6703a926f"
server_key_iv: "133713f15982573ac06346fc2b8f361a"
server_key_version: "encryptionkey@0"
I20260925 22:00:15.044579 23965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "df69e898055146c284d2d61df4cbb54b"
format_stamp: "Formatted at 2026-09-25 22:00:15 on dist-test-slave-2g0f"
server_key: "9c21bb4a7be09f584117b1b6703a926f"
server_key_iv: "133713f15982573ac06346fc2b8f361a"
server_key_version: "encryptionkey@0"
I20260925 22:00:15.046063 23965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 22:00:15.046941 23980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:15.047116 23965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:00:15.047200 23965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "df69e898055146c284d2d61df4cbb54b"
format_stamp: "Formatted at 2026-09-25 22:00:15 on dist-test-slave-2g0f"
server_key: "9c21bb4a7be09f584117b1b6703a926f"
server_key_iv: "133713f15982573ac06346fc2b8f361a"
server_key_version: "encryptionkey@0"
I20260925 22:00:15.047262 23965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:15.061018 23965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:15.063228 23965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:15.063349 23965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:15.067016 23965 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:43999
I20260925 22:00:15.067075 24032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:43999 every 8 connection(s)
I20260925 22:00:15.067404 23965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:15.068316 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:15.070724 24033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b: Bootstrap starting.
I20260925 22:00:15.071311 24033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:15.071587 24033 log.cc:824] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:15.071748 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 23965
I20260925 22:00:15.071894 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:15.072098 2331 external_mini_cluster.cc:1428] Setting key b60b916051cab5726b3d9b9c5a10b845
I20260925 22:00:15.072468 24033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b: No bootstrap required, opened a new log
I20260925 22:00:15.074903 24033 raft_consensus.cc:348] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df69e898055146c284d2d61df4cbb54b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43999 } }
I20260925 22:00:15.075050 24033 raft_consensus.cc:374] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:15.075081 24033 raft_consensus.cc:729] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69e898055146c284d2d61df4cbb54b, State: Initialized, Role: FOLLOWER
I20260925 22:00:15.075213 24033 consensus_queue.cc:260] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df69e898055146c284d2d61df4cbb54b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43999 } }
I20260925 22:00:15.075275 24033 raft_consensus.cc:388] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:15.075295 24033 raft_consensus.cc:482] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:15.075328 24033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:15.075939 24033 raft_consensus.cc:504] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df69e898055146c284d2d61df4cbb54b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43999 } }
I20260925 22:00:15.076030 24033 leader_election.cc:302] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df69e898055146c284d2d61df4cbb54b; no voters:
I20260925 22:00:15.076195 24033 leader_election.cc:288] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:15.076231 24038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:15.076380 24038 raft_consensus.cc:686] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [term 1 LEADER]: Becoming Leader. State: Replica: df69e898055146c284d2d61df4cbb54b, State: Running, Role: LEADER
I20260925 22:00:15.076457 24033 sys_catalog.cc:564] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:15.076463 24038 consensus_queue.cc:237] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df69e898055146c284d2d61df4cbb54b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43999 } }
I20260925 22:00:15.076848 24038 sys_catalog.cc:455] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df69e898055146c284d2d61df4cbb54b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df69e898055146c284d2d61df4cbb54b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43999 } } }
I20260925 22:00:15.076925 24038 sys_catalog.cc:458] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:15.077042 24038 sys_catalog.cc:455] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [sys.catalog]: SysCatalogTable state changed. Reason: New leader df69e898055146c284d2d61df4cbb54b. Latest consensus state: current_term: 1 leader_uuid: "df69e898055146c284d2d61df4cbb54b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df69e898055146c284d2d61df4cbb54b" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 43999 } } }
I20260925 22:00:15.077107 24038 sys_catalog.cc:458] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:15.077414 24045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:15.077926 24045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:15.078276 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:15.072594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59149 (local address 127.2.70.254:43999)
0925 22:00:15.072763 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:15.072770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:15.072799 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:00:15.073705 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:15.073713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:15.073773 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:15.073890 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:15.074140 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:15.075018 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:15.075910 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:15.076438 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:15.076539 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:15.077512 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:15.077516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:15.077517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:15.077735 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:15.077777 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:15.077793 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:15.077851 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:15.078034 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 22:00:15.078064 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260925 22:00:15.079336 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:15.079662 24045 catalog_manager.cc:1347] Generated new cluster ID: 34efb9d13a04494ca3428dda8bcc13c8
I20260925 22:00:15.079751 24045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:15.087164 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:15.084404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42911 (local address 127.2.70.254:43999)
0925 22:00:15.084447 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:15.084449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:15.084606 (+ 157us) server_negotiation.cc:407] Connection header received
0925 22:00:15.084661 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:15.084662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:15.084672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:15.084724 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:15.084857 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:15.085338 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:15.086027 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:15.086236 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:15.086292 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:15.086926 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:15.086927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:15.086928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:15.086932 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:15.086948 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:15.086952 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:15.086999 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:15.087066 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 22:00:15.087076 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 22:00:15.092806 24045 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:15.093293 24045 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:15.101881 24045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df69e898055146c284d2d61df4cbb54b: Generated new TSK 0
I20260925 22:00:15.102077 24045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:15.111492 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 23965
W20260925 22:00:15.115335 23950 connection.cc:535] client connection to 127.2.70.254:43999 recv error: Network error: failed to read from TLS socket (remote: 127.2.70.254:43999): Connection reset by peer (error 104)
2026-09-25T22:00:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1240 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:15 dist-test-slave-2g0f krb5kdc[24064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:15 dist-test-slave-2g0f krb5kdc[24064](info): set up 2 sockets
Sep 25 22:00:15 dist-test-slave-2g0f krb5kdc[24064](info): commencing operation
krb5kdc: starting...
W20260925 22:00:16.075711 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:16 dist-test-slave-2g0f krb5kdc[24064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 22:00:16.194607 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:38253
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:38253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:16.275568 24077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:16.275786 24077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:16.275815 24077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:16.277016 24077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:16.277061 24077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:16.277078 24077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:16.277094 24077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:16.277112 24077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:16.278628 24077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:38253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:38253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:16.278868 24077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:16.279074 24077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:16.283723 24083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:16.283784 24085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:16.283974 24077 server_base.cc:1034] running on GCE node
W20260925 22:00:16.283982 24082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:16.284247 24077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:16.284600 24077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:16.285770 24077 hybrid_clock.cc:648] HybridClock initialized: now 1790373616285752 us; error 29 us; skew 500 ppm
Sep 25 22:00:16 dist-test-slave-2g0f krb5kdc[24064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373616, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:16.288213 24077 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 22:00:16.288614 24077 webserver.cc:460] Webserver started at http://127.2.70.254:35015/ using document root <none> and password file <none>
I20260925 22:00:16.288821 24077 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:16.288868 24077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:16.289052 24077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:16.290102 24077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "fa2dace489e1440f99db568287e2040a"
format_stamp: "Formatted at 2026-09-25 22:00:16 on dist-test-slave-2g0f"
server_key: "97bcd9f42f8078d7fca9d31a3d3435ee"
server_key_iv: "9ab7f234130a6f495b1bbb7a04894cb4"
server_key_version: "encryptionkey@0"
I20260925 22:00:16.290418 24077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "fa2dace489e1440f99db568287e2040a"
format_stamp: "Formatted at 2026-09-25 22:00:16 on dist-test-slave-2g0f"
server_key: "97bcd9f42f8078d7fca9d31a3d3435ee"
server_key_iv: "9ab7f234130a6f495b1bbb7a04894cb4"
server_key_version: "encryptionkey@0"
I20260925 22:00:16.291734 24077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 22:00:16.292536 24092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:16.292686 24077 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 22:00:16.292764 24077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "fa2dace489e1440f99db568287e2040a"
format_stamp: "Formatted at 2026-09-25 22:00:16 on dist-test-slave-2g0f"
server_key: "97bcd9f42f8078d7fca9d31a3d3435ee"
server_key_iv: "9ab7f234130a6f495b1bbb7a04894cb4"
server_key_version: "encryptionkey@0"
I20260925 22:00:16.292822 24077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:16.296056 24077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:16.297825 24077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:16.297955 24077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:16.301599 24077 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:38253
I20260925 22:00:16.301645 24144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:38253 every 8 connection(s)
I20260925 22:00:16.301932 24077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:16.302860 24145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:16.305066 24145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a: Bootstrap starting.
I20260925 22:00:16.305601 24145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:16.305838 24145 log.cc:824] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:16.306528 24145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a: No bootstrap required, opened a new log
I20260925 22:00:16.308161 24145 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2dace489e1440f99db568287e2040a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38253 } }
I20260925 22:00:16.308292 24145 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:16.308313 24145 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa2dace489e1440f99db568287e2040a, State: Initialized, Role: FOLLOWER
I20260925 22:00:16.308441 24145 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2dace489e1440f99db568287e2040a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38253 } }
I20260925 22:00:16.308499 24145 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:16.308521 24145 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:16.308565 24145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:16.309160 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 24077
I20260925 22:00:16.309278 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:16.309448 2331 external_mini_cluster.cc:1428] Setting key bd96f3de05aa52fdd683f930171e1fc4
I20260925 22:00:16.309474 24145 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2dace489e1440f99db568287e2040a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38253 } }
I20260925 22:00:16.309573 24145 leader_election.cc:302] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa2dace489e1440f99db568287e2040a; no voters:
I20260925 22:00:16.309727 24145 leader_election.cc:288] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:16.309749 24148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:16.310039 24145 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:16.310115 24148 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [term 1 LEADER]: Becoming Leader. State: Replica: fa2dace489e1440f99db568287e2040a, State: Running, Role: LEADER
I20260925 22:00:16.310276 24148 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2dace489e1440f99db568287e2040a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38253 } }
I20260925 22:00:16.310679 24148 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa2dace489e1440f99db568287e2040a. Latest consensus state: current_term: 1 leader_uuid: "fa2dace489e1440f99db568287e2040a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2dace489e1440f99db568287e2040a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38253 } } }
I20260925 22:00:16.310691 24149 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa2dace489e1440f99db568287e2040a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2dace489e1440f99db568287e2040a" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38253 } } }
I20260925 22:00:16.310818 24148 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:16.310833 24149 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a [sys.catalog]: This master's current role is: LEADER
W20260925 22:00:16.311775 24165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:00:16.311856 24165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:00:16.312004 24151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:16.309889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48865 (local address 127.2.70.254:38253)
0925 22:00:16.310355 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:16.310366 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:16.310402 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:00:16.310513 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:16.310520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:16.310553 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:16.311021 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:16.311425 (+ 404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:16.311428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:16.311428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:16.311432 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:16.311473 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:16.311487 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:16.311544 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:16.311703 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 22:00:16.311745 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":67,"threads_started":1}
I20260925 22:00:16.312340 24159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:16.312490 24159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:16.312973 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:16.316239 24159 catalog_manager.cc:1347] Generated new cluster ID: 5d2fa9d72f264b75963c076036baa3ec
I20260925 22:00:16.316296 24159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:16.317462 24151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:16.316452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54045 (local address 127.2.70.254:38253)
0925 22:00:16.316483 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:16.316485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:16.316608 (+ 123us) server_negotiation.cc:407] Connection header received
0925 22:00:16.316665 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:16.316666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:16.316671 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:16.316715 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:16.317273 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:16.317274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:16.317274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:16.317278 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:16.317292 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:16.317297 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:16.317345 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:16.317391 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 22:00:16.317404 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 22:00:16.346840 24159 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:16.347008 24159 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:16.351922 24159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa2dace489e1440f99db568287e2040a: Generated new TSK 0
I20260925 22:00:16.352140 24159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:16.357888 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 24077
2026-09-25T22:00:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:16 dist-test-slave-2g0f krb5kdc[24176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:16 dist-test-slave-2g0f krb5kdc[24176](info): set up 2 sockets
Sep 25 22:00:16 dist-test-slave-2g0f krb5kdc[24176](info): commencing operation
krb5kdc: starting...
W20260925 22:00:17.307658 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:17 dist-test-slave-2g0f krb5kdc[24176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 22:00:17.423400 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:46809
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:46809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:17.501642 24189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:17.501845 24189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:17.501875 24189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:17.503026 24189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:17.503065 24189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:17.503085 24189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:17.503100 24189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:17.503116 24189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:17.504578 24189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:46809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:46809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:17.504804 24189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:17.505013 24189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:17.509603 24194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:17.509613 24197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:17.509603 24195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:17.509727 24189 server_base.cc:1034] running on GCE node
I20260925 22:00:17.510075 24189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:17.510313 24189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:17.511457 24189 hybrid_clock.cc:648] HybridClock initialized: now 1790373617511437 us; error 33 us; skew 500 ppm
Sep 25 22:00:17 dist-test-slave-2g0f krb5kdc[24176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373617, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:17.514035 24189 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 22:00:17.514431 24189 webserver.cc:460] Webserver started at http://127.2.70.254:42765/ using document root <none> and password file <none>
I20260925 22:00:17.514627 24189 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:17.514670 24189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:17.514768 24189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:17.515769 24189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "fc60324d39344cdb961b785186d05a01"
format_stamp: "Formatted at 2026-09-25 22:00:17 on dist-test-slave-2g0f"
server_key: "75d29e9e1a62936c06b3d5be74b6493d"
server_key_iv: "fb7230526b0a648ad22c440f04a6df75"
server_key_version: "encryptionkey@0"
I20260925 22:00:17.516140 24189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "fc60324d39344cdb961b785186d05a01"
format_stamp: "Formatted at 2026-09-25 22:00:17 on dist-test-slave-2g0f"
server_key: "75d29e9e1a62936c06b3d5be74b6493d"
server_key_iv: "fb7230526b0a648ad22c440f04a6df75"
server_key_version: "encryptionkey@0"
I20260925 22:00:17.517329 24189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:00:17.518046 24204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:17.518199 24189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:17.518272 24189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "fc60324d39344cdb961b785186d05a01"
format_stamp: "Formatted at 2026-09-25 22:00:17 on dist-test-slave-2g0f"
server_key: "75d29e9e1a62936c06b3d5be74b6493d"
server_key_iv: "fb7230526b0a648ad22c440f04a6df75"
server_key_version: "encryptionkey@0"
I20260925 22:00:17.518340 24189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:17.529567 24189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:17.531376 24189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:17.531497 24189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:17.535115 24189 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:46809
I20260925 22:00:17.535142 24256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:46809 every 8 connection(s)
I20260925 22:00:17.535446 24189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:17.536286 24257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:17.537853 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 24189
I20260925 22:00:17.537964 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:17.538137 2331 external_mini_cluster.cc:1428] Setting key 5ff8b4b43048b9462c99ff945e9c6317
I20260925 22:00:17.538726 24257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01: Bootstrap starting.
I20260925 22:00:17.539284 24257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:17.539585 24257 log.cc:824] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:17.540333 24257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01: No bootstrap required, opened a new log
I20260925 22:00:17.542323 24257 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc60324d39344cdb961b785186d05a01" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46809 } }
I20260925 22:00:17.542477 24257 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:17.542512 24257 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc60324d39344cdb961b785186d05a01, State: Initialized, Role: FOLLOWER
I20260925 22:00:17.542634 24257 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc60324d39344cdb961b785186d05a01" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46809 } }
I20260925 22:00:17.542706 24257 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:17.542737 24257 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:17.542783 24257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:17.543437 24257 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc60324d39344cdb961b785186d05a01" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46809 } }
I20260925 22:00:17.543551 24257 leader_election.cc:302] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc60324d39344cdb961b785186d05a01; no voters:
I20260925 22:00:17.543720 24257 leader_election.cc:288] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:17.543799 24262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:17.544000 24257 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:17.544037 24262 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [term 1 LEADER]: Becoming Leader. State: Replica: fc60324d39344cdb961b785186d05a01, State: Running, Role: LEADER
I20260925 22:00:17.544224 24262 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc60324d39344cdb961b785186d05a01" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46809 } }
I20260925 22:00:17.544584 24262 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc60324d39344cdb961b785186d05a01. Latest consensus state: current_term: 1 leader_uuid: "fc60324d39344cdb961b785186d05a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc60324d39344cdb961b785186d05a01" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46809 } } }
I20260925 22:00:17.544723 24264 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc60324d39344cdb961b785186d05a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc60324d39344cdb961b785186d05a01" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 46809 } } }
I20260925 22:00:17.544787 24262 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:17.544823 24264 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01 [sys.catalog]: This master's current role is: LEADER
Sep 25 22:00:17 dist-test-slave-2g0f krb5kdc[24176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 22:00:17.545047 24271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:17.545542 24271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:17.547247 24271 catalog_manager.cc:1347] Generated new cluster ID: bb3f2e6241d643779d460a5ec3e8d6ce
I20260925 22:00:17.547297 24271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:17.549301 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:17.538609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54017 (local address 127.2.70.254:46809)
0925 22:00:17.538763 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:17.538771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:17.538795 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:00:17.539293 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:17.539301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:17.539351 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:17.539485 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:17.540246 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:17.541202 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:17.542291 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:17.542746 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:17.542863 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:17.545620 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:17.545625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:17.545625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:17.545638 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:17.547680 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:17.548073 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:17.548076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:17.548076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:17.548174 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:17.548336 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:17.548337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:17.548337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:17.548608 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:17.548734 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:17.548973 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 22:00:17.549106 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260925 22:00:17.550541 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:17.564285 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:17.551871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38611 (local address 127.2.70.254:46809)
0925 22:00:17.551913 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:17.551915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:17.552056 (+ 141us) server_negotiation.cc:407] Connection header received
0925 22:00:17.552097 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:17.552097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:17.552107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:17.552174 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:17.552708 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:17.553442 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:17.554371 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:17.554708 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:17.554779 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:17.561570 (+ 6791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:17.561572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:17.561573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:17.561581 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:17.563137 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:17.563504 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:17.563505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:17.563505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:17.563572 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:17.563731 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:17.563732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:17.563732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:17.563905 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:17.564003 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:17.564101 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 22:00:17.564185 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 22:00:17.566682 24271 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:17.567220 24271 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:17.576469 24271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc60324d39344cdb961b785186d05a01: Generated new TSK 0
I20260925 22:00:17.576683 24271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:17.586889 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 24189
2026-09-25T22:00:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1228 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:17 dist-test-slave-2g0f krb5kdc[24288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:17 dist-test-slave-2g0f krb5kdc[24288](info): set up 2 sockets
Sep 25 22:00:17 dist-test-slave-2g0f krb5kdc[24288](info): commencing operation
krb5kdc: starting...
W20260925 22:00:18.539676 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:18 dist-test-slave-2g0f krb5kdc[24288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 22:00:18.653918 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:45211
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:45211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:18.732450 24301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:18.732642 24301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:18.732664 24301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:18.733754 24301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:18.733788 24301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:18.733801 24301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:18.733812 24301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:18.733822 24301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:18.735215 24301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:45211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:45211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:18.735404 24301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:18.735577 24301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:18.740752 24309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:18.740758 24307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:18.740759 24306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:18.741021 24301 server_base.cc:1034] running on GCE node
I20260925 22:00:18.741258 24301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:18.741537 24301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:18.742695 24301 hybrid_clock.cc:648] HybridClock initialized: now 1790373618742659 us; error 57 us; skew 500 ppm
Sep 25 22:00:18 dist-test-slave-2g0f krb5kdc[24288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373618, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:18.745262 24301 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 22:00:18.745708 24301 webserver.cc:460] Webserver started at http://127.2.70.254:36873/ using document root <none> and password file <none>
I20260925 22:00:18.745934 24301 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:18.745982 24301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:18.746105 24301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:18.747149 24301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "575cd078f77b4fbfaf683d59fef61cd3"
format_stamp: "Formatted at 2026-09-25 22:00:18 on dist-test-slave-2g0f"
server_key: "6b6874ed5471a9f27ec5d725410f7c0e"
server_key_iv: "2f890d7e13c078b0ed172e89b10ef107"
server_key_version: "encryptionkey@0"
I20260925 22:00:18.747478 24301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "575cd078f77b4fbfaf683d59fef61cd3"
format_stamp: "Formatted at 2026-09-25 22:00:18 on dist-test-slave-2g0f"
server_key: "6b6874ed5471a9f27ec5d725410f7c0e"
server_key_iv: "2f890d7e13c078b0ed172e89b10ef107"
server_key_version: "encryptionkey@0"
I20260925 22:00:18.748808 24301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 22:00:18.749645 24316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:18.749845 24301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 22:00:18.749987 24301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "575cd078f77b4fbfaf683d59fef61cd3"
format_stamp: "Formatted at 2026-09-25 22:00:18 on dist-test-slave-2g0f"
server_key: "6b6874ed5471a9f27ec5d725410f7c0e"
server_key_iv: "2f890d7e13c078b0ed172e89b10ef107"
server_key_version: "encryptionkey@0"
I20260925 22:00:18.750049 24301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:18.761411 24301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:18.763551 24301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:18.763691 24301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:18.767449 24301 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:45211
I20260925 22:00:18.767484 24368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:45211 every 8 connection(s)
I20260925 22:00:18.767820 24301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:18.768534 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 24301
I20260925 22:00:18.768656 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:18.768853 2331 external_mini_cluster.cc:1428] Setting key 41425ec77e5b83d854effd0f6b255624
I20260925 22:00:18.768939 24369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:18.772688 24369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3: Bootstrap starting.
I20260925 22:00:18.773290 24369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:18.773536 24369 log.cc:824] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:18.774250 24369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3: No bootstrap required, opened a new log
I20260925 22:00:18.775858 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:18.769439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51263 (local address 127.2.70.254:45211)
0925 22:00:18.769645 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:18.769655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:18.769687 (+ 32us) server_negotiation.cc:407] Connection header received
0925 22:00:18.770181 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:18.770191 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:18.770257 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:18.770414 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:18.770644 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:18.772015 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:18.772838 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:18.773295 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:18.773392 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:18.774059 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:18.774062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:18.774063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:18.774217 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:18.774246 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:18.774256 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:18.774312 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:18.774390 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 22:00:18.775493 (+ 1103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260925 22:00:18.775962 24369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575cd078f77b4fbfaf683d59fef61cd3" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 45211 } }
I20260925 22:00:18.776134 24369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:18.776170 24369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575cd078f77b4fbfaf683d59fef61cd3, State: Initialized, Role: FOLLOWER
I20260925 22:00:18.776283 24369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575cd078f77b4fbfaf683d59fef61cd3" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 45211 } }
I20260925 22:00:18.776356 24369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:18.776386 24369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:18.776432 24369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:18.776762 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:18.777065 24369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575cd078f77b4fbfaf683d59fef61cd3" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 45211 } }
I20260925 22:00:18.777151 24369 leader_election.cc:302] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 575cd078f77b4fbfaf683d59fef61cd3; no voters:
I20260925 22:00:18.777297 24369 leader_election.cc:288] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:18.777326 24374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:18.777474 24374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [term 1 LEADER]: Becoming Leader. State: Replica: 575cd078f77b4fbfaf683d59fef61cd3, State: Running, Role: LEADER
I20260925 22:00:18.777539 24369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:18.777601 24374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575cd078f77b4fbfaf683d59fef61cd3" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 45211 } }
I20260925 22:00:18.777966 24375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "575cd078f77b4fbfaf683d59fef61cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575cd078f77b4fbfaf683d59fef61cd3" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 45211 } } }
I20260925 22:00:18.778065 24375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:18.777964 24374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 575cd078f77b4fbfaf683d59fef61cd3. Latest consensus state: current_term: 1 leader_uuid: "575cd078f77b4fbfaf683d59fef61cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575cd078f77b4fbfaf683d59fef61cd3" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 45211 } } }
I20260925 22:00:18.778296 24374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:18.778564 24386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:18.779140 24386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:18.781286 24386 catalog_manager.cc:1347] Generated new cluster ID: 9e422386c41a40f198cac7873cca1481
I20260925 22:00:18.781332 24386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:18.781868 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:18.779001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42221 (local address 127.2.70.254:45211)
0925 22:00:18.779037 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:18.779039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:18.779163 (+ 124us) server_negotiation.cc:407] Connection header received
0925 22:00:18.779384 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:18.779385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:18.779396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:18.779462 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:18.779637 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:18.780124 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:18.780785 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:18.781013 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:18.781070 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:18.781658 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:18.781659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:18.781659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:18.781663 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:18.781677 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:18.781681 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:18.781725 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:18.781772 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 22:00:18.781794 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 22:00:18.788240 24386 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:18.788941 24386 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:18.806170 24386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 575cd078f77b4fbfaf683d59fef61cd3: Generated new TSK 0
I20260925 22:00:18.806391 24386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:18.809864 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 24301
2026-09-25T22:00:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:18 dist-test-slave-2g0f krb5kdc[24400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:18 dist-test-slave-2g0f krb5kdc[24400](info): set up 2 sockets
Sep 25 22:00:18 dist-test-slave-2g0f krb5kdc[24400](info): commencing operation
krb5kdc: starting...
W20260925 22:00:19.767714 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:19 dist-test-slave-2g0f krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 22:00:19.887311 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:40805
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:40805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:19.968487 24413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:19.968703 24413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:19.968726 24413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:19.969900 24413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:19.969946 24413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:19.969959 24413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:19.969970 24413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:19.969981 24413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:19.971582 24413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:40805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:40805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:19.971812 24413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:19.972047 24413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:19.976768 24421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:19.976773 24418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:19.976773 24419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:19.977113 24413 server_base.cc:1034] running on GCE node
I20260925 22:00:19.977358 24413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:19.977625 24413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:19.978767 24413 hybrid_clock.cc:648] HybridClock initialized: now 1790373619978752 us; error 36 us; skew 500 ppm
Sep 25 22:00:19 dist-test-slave-2g0f krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373619, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:19.981173 24413 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 22:00:19.981640 24413 webserver.cc:460] Webserver started at http://127.2.70.254:42047/ using document root <none> and password file <none>
I20260925 22:00:19.981843 24413 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:19.981889 24413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:19.982012 24413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:19.983028 24413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "036b4fb2592449b595a2ddb373273d61"
format_stamp: "Formatted at 2026-09-25 22:00:19 on dist-test-slave-2g0f"
server_key: "ffd8fd97bc4f8c0eae8d5e6a6908bcad"
server_key_iv: "395efdc94862a4cb28512beb420f8073"
server_key_version: "encryptionkey@0"
I20260925 22:00:19.983349 24413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "036b4fb2592449b595a2ddb373273d61"
format_stamp: "Formatted at 2026-09-25 22:00:19 on dist-test-slave-2g0f"
server_key: "ffd8fd97bc4f8c0eae8d5e6a6908bcad"
server_key_iv: "395efdc94862a4cb28512beb420f8073"
server_key_version: "encryptionkey@0"
I20260925 22:00:19.984603 24413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 22:00:19.985278 24428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:19.985433 24413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 22:00:19.985514 24413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "036b4fb2592449b595a2ddb373273d61"
format_stamp: "Formatted at 2026-09-25 22:00:19 on dist-test-slave-2g0f"
server_key: "ffd8fd97bc4f8c0eae8d5e6a6908bcad"
server_key_iv: "395efdc94862a4cb28512beb420f8073"
server_key_version: "encryptionkey@0"
I20260925 22:00:19.985571 24413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:19.988809 24413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:19.990783 24413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:19.990908 24413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:19.994678 24413 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:40805
I20260925 22:00:19.994707 24480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:40805 every 8 connection(s)
I20260925 22:00:19.995010 24413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:19.996132 24481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:19.998360 24481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61: Bootstrap starting.
I20260925 22:00:19.998888 24481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:19.999117 24481 log.cc:824] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:19.999740 24481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61: No bootstrap required, opened a new log
I20260925 22:00:20.001443 24481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036b4fb2592449b595a2ddb373273d61" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40805 } }
I20260925 22:00:20.001587 24481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:00:20.001611 24481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036b4fb2592449b595a2ddb373273d61, State: Initialized, Role: FOLLOWER
I20260925 22:00:20.001765 24481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036b4fb2592449b595a2ddb373273d61" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40805 } }
I20260925 22:00:20.001825 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 24413
I20260925 22:00:20.001852 24481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:00:20.001884 24481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:00:20.001924 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:20.001924 24481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:20.002110 2331 external_mini_cluster.cc:1428] Setting key d5f2d7bd9665a62484a7744043229687
I20260925 22:00:20.002957 24481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036b4fb2592449b595a2ddb373273d61" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40805 } }
I20260925 22:00:20.003073 24481 leader_election.cc:302] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 036b4fb2592449b595a2ddb373273d61; no voters:
I20260925 22:00:20.003274 24481 leader_election.cc:288] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:00:20.003312 24486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:20.003429 24486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [term 1 LEADER]: Becoming Leader. State: Replica: 036b4fb2592449b595a2ddb373273d61, State: Running, Role: LEADER
I20260925 22:00:20.003567 24481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:20.003571 24486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036b4fb2592449b595a2ddb373273d61" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40805 } }
I20260925 22:00:20.004094 24486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 036b4fb2592449b595a2ddb373273d61. Latest consensus state: current_term: 1 leader_uuid: "036b4fb2592449b595a2ddb373273d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036b4fb2592449b595a2ddb373273d61" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40805 } } }
I20260925 22:00:20.004184 24486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:20.004103 24487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "036b4fb2592449b595a2ddb373273d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036b4fb2592449b595a2ddb373273d61" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 40805 } } }
I20260925 22:00:20.004231 24487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:20.004421 24494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:20.004455 24485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:20.002585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39783 (local address 127.2.70.254:40805)
0925 22:00:20.002762 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:20.002770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:20.002795 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:00:20.002897 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:20.002903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:20.002926 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:20.003095 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:20.003753 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:20.003756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:20.003757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:20.003760 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:20.003799 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:20.003812 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:20.003897 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:20.004004 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 22:00:20.004056 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260925 22:00:20.004971 24494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:20.005415 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:20.006600 24494 catalog_manager.cc:1347] Generated new cluster ID: d5a0561ecada4e40991ddf23063477d5
I20260925 22:00:20.006649 24494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:20.013053 24485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:20.008215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48243 (local address 127.2.70.254:40805)
0925 22:00:20.008251 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:20.008253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:20.012060 (+ 3807us) server_negotiation.cc:407] Connection header received
0925 22:00:20.012123 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:20.012124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:20.012142 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:20.012215 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:20.012806 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:20.012807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:20.012808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:00:20.012812 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:20.012830 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:00:20.012834 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:20.012888 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:20.012952 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 22:00:20.012965 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 22:00:20.031854 24494 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:20.032056 24494 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:20.039048 24494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 036b4fb2592449b595a2ddb373273d61: Generated new TSK 0
I20260925 22:00:20.039268 24494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:00:20.049711 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 24413
2026-09-25T22:00:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1239 ms)
[----------] 18 tests from AuthTokenIssuingTest (22269 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:20 dist-test-slave-2g0f krb5kdc[24512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:20 dist-test-slave-2g0f krb5kdc[24512](info): set up 2 sockets
Sep 25 22:00:20 dist-test-slave-2g0f krb5kdc[24512](info): commencing operation
krb5kdc: starting...
W20260925 22:00:20.995937 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 22:00:21.113131 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:38349
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:38349,127.2.70.253:43055,127.2.70.252:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:21.198890 24525 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:00:21.199106 24525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:21.199122 24525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:21.199141 24525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:21.200311 24525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:21.200351 24525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:21.200364 24525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:21.200382 24525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:21.200397 24525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:21.201946 24525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:38349,127.2.70.253:43055,127.2.70.252:43575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:38349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:21.202179 24525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:21.202389 24525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:00:21.207237 24525 server_base.cc:1034] running on GCE node
W20260925 22:00:21.207191 24533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:21.207203 24530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:21.207203 24531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:21.207700 24525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:21.208009 24525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:21.209172 24525 hybrid_clock.cc:648] HybridClock initialized: now 1790373621209156 us; error 29 us; skew 500 ppm
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:21.211606 24525 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 22:00:21.212190 24525 webserver.cc:460] Webserver started at http://127.2.70.254:35437/ using document root <none> and password file <none>
I20260925 22:00:21.212414 24525 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:21.212467 24525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:21.212601 24525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:21.213634 24525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "0b418f74487648e1b9672f7ed69d4700"
format_stamp: "Formatted at 2026-09-25 22:00:21 on dist-test-slave-2g0f"
server_key: "93fe1523dffc18dbd84ce24a03f4c00f"
server_key_iv: "546a44b23f10c086f51447e5bf05f871"
server_key_version: "encryptionkey@0"
I20260925 22:00:21.213966 24525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "0b418f74487648e1b9672f7ed69d4700"
format_stamp: "Formatted at 2026-09-25 22:00:21 on dist-test-slave-2g0f"
server_key: "93fe1523dffc18dbd84ce24a03f4c00f"
server_key_iv: "546a44b23f10c086f51447e5bf05f871"
server_key_version: "encryptionkey@0"
I20260925 22:00:21.215312 24525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 22:00:21.216060 24540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:21.216223 24525 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 22:00:21.216351 24525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "0b418f74487648e1b9672f7ed69d4700"
format_stamp: "Formatted at 2026-09-25 22:00:21 on dist-test-slave-2g0f"
server_key: "93fe1523dffc18dbd84ce24a03f4c00f"
server_key_iv: "546a44b23f10c086f51447e5bf05f871"
server_key_version: "encryptionkey@0"
I20260925 22:00:21.216414 24525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:21.231640 24525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:21.233815 24525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:21.233963 24525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:21.237747 24525 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:38349
I20260925 22:00:21.237787 24592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:38349 every 8 connection(s)
I20260925 22:00:21.238121 24525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:21.239104 24593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:21.240679 24593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 22:00:21.248111 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 24525
I20260925 22:00:21.248219 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
I20260925 22:00:21.248413 2331 external_mini_cluster.cc:1428] Setting key b9d43f09f5d632f1f266c86029deea25
I20260925 22:00:21.250422 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.241365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.2.70.254:38349)
0925 22:00:21.241530 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.241539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.241647 (+ 108us) server_negotiation.cc:407] Connection header received
0925 22:00:21.242231 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.242233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.242249 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.242328 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.242875 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.243681 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.244552 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.244850 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.244945 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.247038 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.247045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.247046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.247053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.248794 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.249219 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.249222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.249222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.249323 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.249529 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.249529 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.249530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.249821 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.249893 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.250154 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.250253 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260925 22:00:21.250453 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.241322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:38349 (local address 127.0.0.1:49022)
0925 22:00:21.241518 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.241562 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:21.242080 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:21.242348 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:21.242350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:21.242627 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:21.242788 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.242793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.243764 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.243765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.244479 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.244482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.244943 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.244944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.245007 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.245495 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:21.245503 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:21.246951 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:21.248892 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.248899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.248900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.249132 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.249387 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.249388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.249388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.249473 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.249909 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:21.249916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:21.249998 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 22:00:21.250117 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 22:00:21.250345 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":60,"thread_start_us":88,"threads_started":1}
I20260925 22:00:21.251210 24593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:21.251546 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.251411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43055 (local address 127.0.0.1:42700)
0925 22:00:21.251451 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.251523 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:43055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 22:00:21.251680 24541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.253:43055: connect: Connection refused (error 111)
I20260925 22:00:21.252118 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.252024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43055 (local address 127.0.0.1:42708)
0925 22:00:21.252067 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.252097 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:43055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 22:00:21.252220 24593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.253:43055: Network error: Client connection negotiation failed: client connection to 127.2.70.253:43055: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.70.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.253.keytab.
Entry for principal kudu/127.2.70.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.253.keytab.
I20260925 22:00:21.274771 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.253:43055
--webserver_interface=127.2.70.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.253.keytab
--principal=kudu/127.2.70.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:38349,127.2.70.253:43055,127.2.70.252:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:00:21.301452 24593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } attempt: 1
I20260925 22:00:21.301967 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.301826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43055 (local address 127.0.0.1:42716)
0925 22:00:21.301885 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.301932 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:43055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 22:00:21.302260 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.302178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43055 (local address 127.0.0.1:42726)
0925 22:00:21.302215 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.302241 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:43055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 22:00:21.302419 24593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.253:43055: Network error: Client connection negotiation failed: client connection to 127.2.70.253:43055: connect: Connection refused (error 111)
W20260925 22:00:21.354561 24600 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:00:21.354784 24600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:21.354800 24600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:21.354820 24600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:21.356086 24600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:21.356135 24600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:21.356149 24600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:21.356160 24600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:21.356172 24600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:21.357735 24600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:38349,127.2.70.253:43055,127.2.70.252:43575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.253.keytab
--principal=kudu/127.2.70.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.253:43055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.70.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:21.357976 24600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:21.358201 24600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:21.362890 24605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:21.362854 24608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:21.362900 24606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:21.363106 24600 server_base.cc:1034] running on GCE node
I20260925 22:00:21.363420 24600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:21.363694 24600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:21.364822 24600 hybrid_clock.cc:648] HybridClock initialized: now 1790373621364812 us; error 26 us; skew 500 ppm
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:21.367156 24600 init.cc:377] Logged in from keytab as kudu/127.2.70.253@KRBTEST.COM (short username kudu)
I20260925 22:00:21.367586 24600 webserver.cc:460] Webserver started at http://127.2.70.253:39655/ using document root <none> and password file <none>
I20260925 22:00:21.367786 24600 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:21.367848 24600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:21.367978 24600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:21.369112 24600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/data/instance:
uuid: "757c2d411fd743cd846bcbde378d295d"
format_stamp: "Formatted at 2026-09-25 22:00:21 on dist-test-slave-2g0f"
server_key: "5914a4cab00941f11b884456194b8850"
server_key_iv: "e44e483c7f641049c661de0c70e24145"
server_key_version: "encryptionkey@0"
I20260925 22:00:21.369436 24600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/wal/instance:
uuid: "757c2d411fd743cd846bcbde378d295d"
format_stamp: "Formatted at 2026-09-25 22:00:21 on dist-test-slave-2g0f"
server_key: "5914a4cab00941f11b884456194b8850"
server_key_iv: "e44e483c7f641049c661de0c70e24145"
server_key_version: "encryptionkey@0"
I20260925 22:00:21.370613 24593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } attempt: 2
I20260925 22:00:21.370697 24600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 22:00:21.371119 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.370979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43055 (local address 127.0.0.1:42730)
0925 22:00:21.371047 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.371085 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:43055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:00:21.371450 24615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:21.371464 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.371328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43055 (local address 127.0.0.1:42736)
0925 22:00:21.371395 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.371432 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:43055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:00:21.371562 24593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.253:43055: Network error: Client connection negotiation failed: client connection to 127.2.70.253:43055: connect: Connection refused (error 111)
I20260925 22:00:21.371675 24600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:00:21.371757 24600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/wal
uuid: "757c2d411fd743cd846bcbde378d295d"
format_stamp: "Formatted at 2026-09-25 22:00:21 on dist-test-slave-2g0f"
server_key: "5914a4cab00941f11b884456194b8850"
server_key_iv: "e44e483c7f641049c661de0c70e24145"
server_key_version: "encryptionkey@0"
I20260925 22:00:21.371814 24600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:21.395579 24600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:21.397888 24600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:21.398015 24600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:21.402155 24600 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.253:43055
I20260925 22:00:21.402204 24667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.253:43055 every 8 connection(s)
I20260925 22:00:21.402549 24600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
I20260925 22:00:21.403525 24668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:21.405161 24668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:21.409833 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 24600
I20260925 22:00:21.409946 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-1/wal/instance
I20260925 22:00:21.410130 2331 external_mini_cluster.cc:1428] Setting key 733e8ee09a236bdb31a26e7c3361a27a
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.253@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 22:00:21.414333 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.405821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49034 (local address 127.2.70.254:38349)
0925 22:00:21.405874 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.405876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.406125 (+ 249us) server_negotiation.cc:407] Connection header received
0925 22:00:21.406604 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.406606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.406619 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.406670 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.407299 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.407793 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.408784 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.408997 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.409053 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.411567 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.411569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.411569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.411578 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.413021 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.413460 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.413461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.413462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.413517 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.413734 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.413735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.413735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.413849 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.413902 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.414163 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.414243 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 22:00:21.414499 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.405785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:38349 (local address 127.0.0.1:49034)
0925 22:00:21.405995 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.406032 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:21.406503 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:21.406747 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:21.406750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:21.407002 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:21.407201 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.407205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.407911 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.407912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.408704 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.408706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.409085 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.409086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.409193 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.409676 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:21.409687 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:21.411484 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:21.413108 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.413115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.413117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.413388 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.413571 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.413572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.413572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.413679 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.413912 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:21.413918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:21.414049 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 22:00:21.414135 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:00:21.414286 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260925 22:00:21.415025 24668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.70.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.252@KRBTEST.COM" created.
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.253@KRBTEST.COM for kudu/127.2.70.253@KRBTEST.COM
I20260925 22:00:21.422973 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.415247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43055 (local address 127.0.0.1:42750)
0925 22:00:21.415286 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.415296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:21.415333 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:21.415581 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:21.415581 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:21.415704 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:21.415747 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.415749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.416580 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.416581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.417116 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.417118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.417562 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.417563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.417622 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.417984 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:21.417988 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:21.419960 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:21.421825 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.421826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.421826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.422037 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.422252 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.422252 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.422253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.422316 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.422636 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:21.422637 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:21.422688 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 22:00:21.422750 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 22:00:21.422833 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 22:00:21.423038 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.415319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.2.70.253:43055)
0925 22:00:21.415460 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.415464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.415480 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:00:21.415498 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.415499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.415511 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.415569 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.415808 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.416502 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.417182 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.417505 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.417567 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.420055 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.420063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.420063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.420070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.421743 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.422108 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.422111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.422111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.422170 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.422376 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.422377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.422377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.422555 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.422638 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.422813 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.422891 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260925 22:00:21.423658 24668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:21.424031 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.423908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40044)
0925 22:00:21.423954 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.424010 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 22:00:21.424168 24618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
I20260925 22:00:21.424556 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.424488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40048)
0925 22:00:21.424515 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.424540 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 22:00:21.424656 24668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:43575: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.252.keytab.
Entry for principal kudu/127.2.70.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.252.keytab.
I20260925 22:00:21.436034 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.252:43575
--webserver_interface=127.2.70.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.252.keytab
--principal=kudu/127.2.70.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:38349,127.2.70.253:43055,127.2.70.252:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:00:21.462726 24593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } attempt: 3
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for kudu/127.2.70.253@KRBTEST.COM
I20260925 22:00:21.469548 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.463098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43055 (local address 127.0.0.1:42766)
0925 22:00:21.463164 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.463175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:21.463223 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:21.463351 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:21.463352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:21.463522 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:21.463585 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.463587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.464340 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.464341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.464874 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.464876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.465237 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.465238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.465308 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.465733 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:21.465738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:21.467123 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:21.468640 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.468641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.468642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.468803 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.468980 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.468981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.468981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.469037 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.469253 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:21.469254 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:21.469302 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 22:00:21.469379 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:00:21.469477 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:00:21.469623 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.463169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.2.70.253:43055)
0925 22:00:21.463228 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.463230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.463255 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:00:21.463283 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.463284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.463294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.463336 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.463663 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.464263 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.464948 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.465175 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.465230 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.467203 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.467204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.467204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.467212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.468570 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.468865 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.468866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.468866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.468928 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.469087 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.469087 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.469088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.469201 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.469256 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.469440 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.469531 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:00:21.469978 24593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:21.470300 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.470207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40064)
0925 22:00:21.470249 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.470276 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 22:00:21.470635 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.470547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40070)
0925 22:00:21.470583 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.470617 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 22:00:21.470710 24593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:43575: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
I20260925 22:00:21.473881 24668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } attempt: 1
I20260925 22:00:21.474289 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.474181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40084)
0925 22:00:21.474227 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.474255 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 22:00:21.474656 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.474561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40094)
0925 22:00:21.474598 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.474634 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 22:00:21.474747 24668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:43575: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
I20260925 22:00:21.501848 24593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } attempt: 1
I20260925 22:00:21.502365 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.502205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40102)
0925 22:00:21.502290 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.502333 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 22:00:21.502763 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.502668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40108)
0925 22:00:21.502718 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.502742 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 22:00:21.502861 24593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:43575: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
W20260925 22:00:21.516255 24675 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:00:21.516487 24675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:21.516512 24675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:21.516541 24675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:21.517692 24675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:21.517733 24675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:21.517752 24675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:21.517767 24675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:21.517783 24675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:21.519362 24675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:38349,127.2.70.253:43055,127.2.70.252:43575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.252.keytab
--principal=kudu/127.2.70.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.252:43575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.70.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:21.519598 24675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:21.519804 24675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:21.524565 24680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:21.524626 24675 server_base.cc:1034] running on GCE node
W20260925 22:00:21.524573 24683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:21.524573 24681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:21.525000 24675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:21.525295 24675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:21.526467 24675 hybrid_clock.cc:648] HybridClock initialized: now 1790373621526448 us; error 34 us; skew 500 ppm
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:21.529062 24675 init.cc:377] Logged in from keytab as kudu/127.2.70.252@KRBTEST.COM (short username kudu)
I20260925 22:00:21.529508 24675 webserver.cc:460] Webserver started at http://127.2.70.252:46535/ using document root <none> and password file <none>
I20260925 22:00:21.529718 24675 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:21.529762 24675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:21.529882 24675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:21.530962 24675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/data/instance:
uuid: "79451f87936e4e08bf52a51214b570b4"
format_stamp: "Formatted at 2026-09-25 22:00:21 on dist-test-slave-2g0f"
server_key: "91faec2ca7ca493afec314f04044a86c"
server_key_iv: "99be8096998d9d8460030749d7805999"
server_key_version: "encryptionkey@0"
I20260925 22:00:21.531299 24675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/wal/instance:
uuid: "79451f87936e4e08bf52a51214b570b4"
format_stamp: "Formatted at 2026-09-25 22:00:21 on dist-test-slave-2g0f"
server_key: "91faec2ca7ca493afec314f04044a86c"
server_key_iv: "99be8096998d9d8460030749d7805999"
server_key_version: "encryptionkey@0"
I20260925 22:00:21.532694 24675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 22:00:21.533434 24690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:21.533598 24675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 22:00:21.533687 24675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/wal
uuid: "79451f87936e4e08bf52a51214b570b4"
format_stamp: "Formatted at 2026-09-25 22:00:21 on dist-test-slave-2g0f"
server_key: "91faec2ca7ca493afec314f04044a86c"
server_key_iv: "99be8096998d9d8460030749d7805999"
server_key_version: "encryptionkey@0"
I20260925 22:00:21.533757 24675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:21.542953 24668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } attempt: 2
I20260925 22:00:21.543478 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.543326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40122)
0925 22:00:21.543390 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.543438 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 22:00:21.543862 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.543768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40134)
0925 22:00:21.543799 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.543822 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 22:00:21.544001 24668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:43575: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43575: connect: Connection refused (error 111)
I20260925 22:00:21.559657 24675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:21.561748 24675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:21.561857 24675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:21.565563 24675 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.252:43575
I20260925 22:00:21.565609 24742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.252:43575 every 8 connection(s)
I20260925 22:00:21.565965 24675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
I20260925 22:00:21.566972 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:21.568572 24743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:21.570986 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 24675
I20260925 22:00:21.571084 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790373357410105-2331-0/minicluster-data/master-2/wal/instance
I20260925 22:00:21.571237 2331 external_mini_cluster.cc:1428] Setting key bbd0c6068de06310d4e93eda6a6e8246
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.252@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.252@KRBTEST.COM
I20260925 22:00:21.577929 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.569258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.2.70.254:38349)
0925 22:00:21.569308 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.569310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.569523 (+ 213us) server_negotiation.cc:407] Connection header received
0925 22:00:21.570012 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.570013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.570026 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.570080 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.570715 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.571226 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.572393 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.572593 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.572650 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.574805 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.574807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.574808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.574816 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.576402 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.576899 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.576900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.576900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.577004 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.577271 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.577272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.577272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.577420 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.577484 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.577778 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.577859 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 22:00:21.578127 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.569227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:38349 (local address 127.0.0.1:49050)
0925 22:00:21.569382 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.569418 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:21.569903 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:21.570155 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:21.570159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:21.570416 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:21.570629 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.570633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.571302 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.571303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.572309 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.572312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.572679 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.572680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.572785 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.573258 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:21.573266 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:21.574712 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:21.576486 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.576493 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.576495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.576784 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.577078 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.577079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.577079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.577210 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.577489 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:21.577496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:21.577648 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 22:00:21.577734 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:00:21.577885 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260925 22:00:21.578094 24593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } attempt: 2
I20260925 22:00:21.578768 24743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:21.580020 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.571624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.2.70.252:43575)
0925 22:00:21.571782 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.571786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.571859 (+ 73us) server_negotiation.cc:407] Connection header received
0925 22:00:21.571940 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.571946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.571969 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.572032 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.572614 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.573295 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.574037 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.574437 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.574496 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.577076 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.577085 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.577086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.577093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.578859 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.579190 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.579193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.579194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.579252 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.579447 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.579447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.579447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.579607 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.579717 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.579818 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.579922 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for kudu/127.2.70.252@KRBTEST.COM
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.252@KRBTEST.COM for kudu/127.2.70.253@KRBTEST.COM
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 22:00:21.586844 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.579122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.2.70.253:43055)
0925 22:00:21.579168 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.579170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.579981 (+ 811us) server_negotiation.cc:407] Connection header received
0925 22:00:21.580079 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.580080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.580093 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.580121 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.580410 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.581096 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.582017 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.582246 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.582322 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.584361 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.584363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.584363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.584371 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.585681 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.586033 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.586035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.586035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.586090 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.586271 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.586272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.586273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.586427 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.586504 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.586689 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.586776 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 22:00:21.587150 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.579050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:43055 (local address 127.0.0.1:42770)
0925 22:00:21.579935 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.579945 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:21.580007 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:21.580175 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:21.580176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:21.580277 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:21.580329 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.580331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.581177 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.581178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.581905 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.581908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.582329 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.582330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.582406 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.582830 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:21.582835 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:21.584284 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:21.585751 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.585752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.585753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.585949 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.586160 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.586161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.586161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.586214 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.586511 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:21.586512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:21.586560 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 22:00:21.586964 (+ 404us) client_negotiation.cc:240] Negotiation successful
0925 22:00:21.587066 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870}
I20260925 22:00:21.587433 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.578405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40146)
0925 22:00:21.578456 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.578465 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:21.578501 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:21.578835 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:21.578836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:21.578980 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:21.579027 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.579029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.579916 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.579918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.580682 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.580685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.581101 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.581101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.581157 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.582218 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:21.582222 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:21.583643 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:21.586008 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.586009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.586009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.586210 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.586663 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.586663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.586664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.586731 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.587121 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:21.587122 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:21.587167 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 22:00:21.587214 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 22:00:21.587333 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 22:00:21.587530 24743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:21.587778 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.578568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.2.70.252:43575)
0925 22:00:21.578727 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.578729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.578736 (+ 7us) server_negotiation.cc:407] Connection header received
0925 22:00:21.578752 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.578753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.578762 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.578821 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.579110 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.579856 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.580758 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.581044 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.581108 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.583725 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.583727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.583727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.583733 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.585895 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.586534 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.586535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.586535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.586590 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.586803 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.586804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.586804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.587004 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.587549 (+ 545us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.587588 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.587661 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260925 22:00:21.588804 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.580909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.2.70.254:38349)
0925 22:00:21.580943 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.580944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.580953 (+ 9us) server_negotiation.cc:407] Connection header received
0925 22:00:21.580979 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.580980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.580987 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.581033 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.581527 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.582136 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.583077 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.583272 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.583333 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.585810 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.585812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.585812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.585819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.587493 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.587883 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.587885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.587885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.587960 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.588158 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.588159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.588159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.588282 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.588417 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.588631 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.588709 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 22:00:21.589383 24593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700: Bootstrap starting.
I20260925 22:00:21.590183 24593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:21.590456 24593 log.cc:824] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.252@KRBTEST.COM for kudu/127.2.70.252@KRBTEST.COM
I20260925 22:00:21.591280 24593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700: No bootstrap required, opened a new log
I20260925 22:00:21.593170 24593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } }
I20260925 22:00:21.593322 24593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b418f74487648e1b9672f7ed69d4700, State: Initialized, Role: FOLLOWER
I20260925 22:00:21.593431 24593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } }
I20260925 22:00:21.593770 24593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:21.594095 24750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } } }
I20260925 22:00:21.594259 24750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:00:21.594709 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.587793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40152)
0925 22:00:21.587879 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.587888 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:21.587920 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:21.588101 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:21.588102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:21.588242 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:21.588305 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.588308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.589082 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.589083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.589624 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.589626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.589955 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.589956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.589998 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.590328 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:21.590332 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:21.592130 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:21.593666 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.593667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.593667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.593862 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.594062 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.594063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.594063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.594117 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.594417 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:21.594417 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:21.594453 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:00:21.594510 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 22:00:21.594607 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 22:00:21.594751 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.587887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.2.70.252:43575)
0925 22:00:21.588019 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.588021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.588027 (+ 6us) server_negotiation.cc:407] Connection header received
0925 22:00:21.588041 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.588041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.588049 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.588091 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.588358 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.589019 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.589687 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.589903 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.589954 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.592223 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.592224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.592225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.592231 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.593609 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.593925 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.593926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.593926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.593994 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.594170 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.594171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.594171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.594337 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.594397 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.594531 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.594653 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":118}
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.253@KRBTEST.COM
I20260925 22:00:21.596431 24743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4: Bootstrap starting.
I20260925 22:00:21.597114 24743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:21.597385 24743 log.cc:824] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4: Log is configured to *not* fsync() on all Append() calls
W20260925 22:00:21.595825 24761 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:00:21.597720 24761 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:00:21.598241 24743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4: No bootstrap required, opened a new log
I20260925 22:00:21.599197 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.591206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.2.70.253:43055)
0925 22:00:21.591238 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.591240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.591246 (+ 6us) server_negotiation.cc:407] Connection header received
0925 22:00:21.591286 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.591287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.591296 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.591340 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.591752 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.592311 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.593250 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.593562 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.593640 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.596245 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.596246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.596247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.596255 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.597559 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.598104 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.598105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.598106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.598163 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.598353 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.598354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.598354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.598497 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.598605 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.599015 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.599097 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 22:00:21.600319 24743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } }
I20260925 22:00:21.600476 24743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79451f87936e4e08bf52a51214b570b4, State: Initialized, Role: FOLLOWER
I20260925 22:00:21.600606 24743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } }
I20260925 22:00:21.600920 24743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:21.598737 24525 master_runner.cc:186] Error getting master registration for 127.2.70.253:43055: OK, instance_id { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" instance_seqno: 1790373621401025 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:00:21.601198 24525 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:00:21.602710 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 22:00:21.602782 24775 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:00:21.602818 24775 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:00:21.603698 24567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0b418f74487648e1b9672f7ed69d4700"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49054
I20260925 22:00:21.603811 24567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:00:21.603940 24567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:21.605288 24567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } }
I20260925 22:00:21.605495 24567 leader_election.cc:288] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [CANDIDATE]: Term 1 election: Requested vote from peers 757c2d411fd743cd846bcbde378d295d (127.2.70.253:43055), 79451f87936e4e08bf52a51214b570b4 (127.2.70.252:43575)
I20260925 22:00:21.605773 24717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b418f74487648e1b9672f7ed69d4700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "79451f87936e4e08bf52a51214b570b4"
I20260925 22:00:21.605878 24717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:21.603894 24675 master_runner.cc:186] Error getting master registration for 127.2.70.253:43055: OK, instance_id { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" instance_seqno: 1790373621401025 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:00:21.606088 24675 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:00:21.600878 24764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } } }
I20260925 22:00:21.606148 24764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:00:21.606760 24717 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b418f74487648e1b9672f7ed69d4700 in term 1.
I20260925 22:00:21.606987 24543 leader_election.cc:302] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b418f74487648e1b9672f7ed69d4700, 79451f87936e4e08bf52a51214b570b4; no voters:
I20260925 22:00:21.607106 24750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:21.607270 24750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [term 1 LEADER]: Becoming Leader. State: Replica: 0b418f74487648e1b9672f7ed69d4700, State: Running, Role: LEADER
I20260925 22:00:21.607383 24750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } }
I20260925 22:00:21.607791 24750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b418f74487648e1b9672f7ed69d4700. Latest consensus state: current_term: 1 leader_uuid: "0b418f74487648e1b9672f7ed69d4700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } } }
I20260925 22:00:21.607890 24750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:21.608093 24783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:21.608441 24783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:21.617007 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.608191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.2.70.254:38349)
0925 22:00:21.608232 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.608234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.608658 (+ 424us) server_negotiation.cc:407] Connection header received
0925 22:00:21.608757 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.608758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.608769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.608796 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.609728 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.610492 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.611665 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.612096 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.612158 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.613687 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.613689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.613689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.613696 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.615328 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.615672 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.615673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.615674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.615734 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.616172 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.616173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.616174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.616317 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.616422 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.616814 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.616896 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 22:00:21.618556 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.609912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.2.70.252:43575)
0925 22:00:21.609949 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.609951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.610141 (+ 190us) server_negotiation.cc:407] Connection header received
0925 22:00:21.610231 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.610232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.610242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.610271 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.610939 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.611710 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.612604 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.612923 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.612992 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.615721 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.615723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.615723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.615730 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.617370 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.617751 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.617752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.617753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.617810 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.618007 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.618008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.618008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.618147 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.618252 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.618373 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.618463 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 22:00:21.624639 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.608495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.2.70.253:43055)
0925 22:00:21.608538 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.608540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.609628 (+ 1088us) server_negotiation.cc:407] Connection header received
0925 22:00:21.609690 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.609691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.609702 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.609749 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.610574 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.611402 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.619941 (+ 8539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.620302 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.620346 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.621889 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.621890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.621891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.621899 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.623466 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.623819 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.623820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.623821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.623898 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.624101 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.624101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.624102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.624236 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.624331 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.624447 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.624538 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 22:00:21.626025 24717 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [term 1 FOLLOWER]: Refusing update from remote peer 0b418f74487648e1b9672f7ed69d4700: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:00:21.626385 24750 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:00:21.627815 24764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b418f74487648e1b9672f7ed69d4700. Latest consensus state: current_term: 1 leader_uuid: "0b418f74487648e1b9672f7ed69d4700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } } }
I20260925 22:00:21.627948 24764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:00:21.629294 24750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b418f74487648e1b9672f7ed69d4700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } } }
I20260925 22:00:21.629390 24750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:21.630151 24764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b418f74487648e1b9672f7ed69d4700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } } }
I20260925 22:00:21.630235 24764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:00:21.630527 24788 mvcc.cc:205] Tried to move back new op lower bound from 7333370354176712704 to 7333370354104578048. Current Snapshot: MvccSnapshot[applied={T|T < 7333370354176712704}]
I20260925 22:00:21.635138 24668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } attempt: 3
Sep 25 22:00:21 dist-test-slave-2g0f krb5kdc[24512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.253@KRBTEST.COM for kudu/127.2.70.252@KRBTEST.COM
I20260925 22:00:21.640319 24783 catalog_manager.cc:1347] Generated new cluster ID: 63dda9467b6f4644a36c82b2bd8fad32
I20260925 22:00:21.640373 24783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:21.643954 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.635410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43575 (local address 127.0.0.1:40176)
0925 22:00:21.635457 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:21.635468 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:21.635504 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:21.635634 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:21.635635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:21.635822 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:21.635924 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.635927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.636854 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.636855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.637757 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:21.637761 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.638575 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.638576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:21.638640 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.639115 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:21.639120 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:21.640616 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:21.642452 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.642453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.642454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.642667 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.643251 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:21.643251 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:21.643252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:21.643324 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:21.643608 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:21.643609 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:21.643655 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 22:00:21.643714 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 22:00:21.643815 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 22:00:21.644304 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:21.635469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.2.70.252:43575)
0925 22:00:21.635508 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.635510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.635542 (+ 32us) server_negotiation.cc:407] Connection header received
0925 22:00:21.635563 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.635563 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.635573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.635621 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.636022 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.636778 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.637854 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:21.638184 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:21.638241 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:21.640710 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:21.640712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:21.640712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:21.640720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:21.642363 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.642749 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.642750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.642751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.642822 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:21.643389 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:21.643390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:21.643391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:21.643529 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:21.643583 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:21.644134 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 22:00:21.644213 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 22:00:21.646312 24668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d: Bootstrap starting.
I20260925 22:00:21.647059 24668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:21.647352 24668 log.cc:824] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:21.651875 24668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d: No bootstrap required, opened a new log
I20260925 22:00:21.653874 24668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } }
I20260925 22:00:21.654031 24668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757c2d411fd743cd846bcbde378d295d, State: Initialized, Role: FOLLOWER
I20260925 22:00:21.654114 24668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } }
I20260925 22:00:21.654434 24668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:21.654470 24642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b418f74487648e1b9672f7ed69d4700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "757c2d411fd743cd846bcbde378d295d"
I20260925 22:00:21.654592 24642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:21.655468 24642 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b418f74487648e1b9672f7ed69d4700 in term 1.
I20260925 22:00:21.655794 24641 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [term 1 FOLLOWER]: Refusing update from remote peer 0b418f74487648e1b9672f7ed69d4700: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:00:21.656140 24777 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [LEADER]: Connected to new peer: Peer: permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:00:21.656994 24804 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:00:21.657050 24804 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:00:21.660084 24793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b418f74487648e1b9672f7ed69d4700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } } }
I20260925 22:00:21.660172 24793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:00:21.660223 24793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b418f74487648e1b9672f7ed69d4700. Latest consensus state: current_term: 1 leader_uuid: "0b418f74487648e1b9672f7ed69d4700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } } }
I20260925 22:00:21.660269 24793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:00:21.660311 24793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b418f74487648e1b9672f7ed69d4700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } } }
I20260925 22:00:21.660354 24793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:00:21.666615 24783 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:21.667358 24783 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:21.668259 24807 mvcc.cc:205] Tried to move back new op lower bound from 7333370354176712704 to 7333370354104578048. Current Snapshot: MvccSnapshot[applied={T|T < 7333370354176712704}]
I20260925 22:00:21.669023 24750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b418f74487648e1b9672f7ed69d4700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b418f74487648e1b9672f7ed69d4700" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 38349 } } peers { permanent_uuid: "757c2d411fd743cd846bcbde378d295d" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 43055 } } peers { permanent_uuid: "79451f87936e4e08bf52a51214b570b4" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43575 } } }
I20260925 22:00:21.669123 24750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:21.678889 24783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b418f74487648e1b9672f7ed69d4700: Generated new TSK 0
I20260925 22:00:21.679116 24783 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:00:21.702813 24747 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:21.701017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.2.70.252:43575)
0925 22:00:21.701065 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.701067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.702079 (+ 1012us) server_negotiation.cc:407] Connection header received
0925 22:00:21.702184 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.702185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.702195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.702226 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.702743 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40190: BlockingRecv error: recv got EOF from 127.0.0.1:40190 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 22:00:21.703730 24596 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:21.701487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.2.70.254:38349)
0925 22:00:21.701529 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.701531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.703205 (+ 1674us) server_negotiation.cc:407] Connection header received
0925 22:00:21.703299 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.703300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.703311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.703338 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.703666 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49076: BlockingRecv error: recv got EOF from 127.0.0.1:49076 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 22:00:21.704639 24672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:21.701811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.2.70.253:43055)
0925 22:00:21.701851 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.701853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.704067 (+ 2214us) server_negotiation.cc:407] Connection header received
0925 22:00:21.704194 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.704196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.704207 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.704235 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.704575 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42798: BlockingRecv error: recv got EOF from 127.0.0.1:42798 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 22:00:21.707173 24672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:21.706343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.2.70.253:43055)
0925 22:00:21.706378 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.706380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.706655 (+ 275us) server_negotiation.cc:407] Connection header received
0925 22:00:21.706763 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.706764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.706783 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.706811 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.707125 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42812: BlockingRecv error: recv got EOF from 127.0.0.1:42812 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 22:00:21.710237 24672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:21.709507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.2.70.253:43055)
0925 22:00:21.709539 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.709541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.709751 (+ 210us) server_negotiation.cc:407] Connection header received
0925 22:00:21.709843 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.709844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.709854 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.709882 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.710192 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42818: BlockingRecv error: recv got EOF from 127.0.0.1:42818 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 22:00:21.714229 24672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:21.713483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.2.70.253:43055)
0925 22:00:21.713515 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.713517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.713724 (+ 207us) server_negotiation.cc:407] Connection header received
0925 22:00:21.713814 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.713815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.713826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.713855 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.714181 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42828: BlockingRecv error: recv got EOF from 127.0.0.1:42828 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 22:00:21.720636 24672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:21.719680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.2.70.253:43055)
0925 22:00:21.719722 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.719724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.720074 (+ 350us) server_negotiation.cc:407] Connection header received
0925 22:00:21.720171 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.720173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.720184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.720213 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.720582 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42832: BlockingRecv error: recv got EOF from 127.0.0.1:42832 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 22:00:21.731211 24672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:21.730184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.2.70.253:43055)
0925 22:00:21.730228 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.730230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.730623 (+ 393us) server_negotiation.cc:407] Connection header received
0925 22:00:21.730729 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.730731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.730743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.730771 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.731149 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42848: BlockingRecv error: recv got EOF from 127.0.0.1:42848 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 22:00:21.749722 24672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:21.748808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.2.70.253:43055)
0925 22:00:21.748854 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.748856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.749109 (+ 253us) server_negotiation.cc:407] Connection header received
0925 22:00:21.749211 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.749212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.749225 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.749256 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.749661 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42858: BlockingRecv error: recv got EOF from 127.0.0.1:42858 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 22:00:21.783877 24672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:21.783159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.2.70.253:43055)
0925 22:00:21.783203 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.783205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.783327 (+ 122us) server_negotiation.cc:407] Connection header received
0925 22:00:21.783379 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.783380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.783413 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.783461 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.783785 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42862: BlockingRecv error: recv got EOF from 127.0.0.1:42862 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 22:00:21.849844 24672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:21.849208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.2.70.253:43055)
0925 22:00:21.849281 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.849283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.849337 (+ 54us) server_negotiation.cc:407] Connection header received
0925 22:00:21.849383 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.849384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.849394 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.849474 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.849795 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42868: BlockingRecv error: recv got EOF from 127.0.0.1:42868 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 22:00:21.979599 24672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:21.979035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.2.70.253:43055)
0925 22:00:21.979095 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:21.979097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:21.979124 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:00:21.979201 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:21.979203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:21.979212 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:21.979261 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:21.979535 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42872: BlockingRecv error: recv got EOF from 127.0.0.1:42872 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 22:00:22.237654 24672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:22.236946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.2.70.253:43055)
0925 22:00:22.237019 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:22.237021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:22.237031 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:00:22.237108 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:22.237109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:22.237122 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:22.237187 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:22.237590 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42882: BlockingRecv error: recv got EOF from 127.0.0.1:42882 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:00:22.603256 24775 catalog_manager.cc:1259] Loaded cluster ID: 63dda9467b6f4644a36c82b2bd8fad32
I20260925 22:00:22.603341 24775 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4: loading cluster ID for follower catalog manager: success
I20260925 22:00:22.604193 24775 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4: acquiring CA information for follower catalog manager: success
I20260925 22:00:22.604476 24775 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79451f87936e4e08bf52a51214b570b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:00:22.657469 24804 catalog_manager.cc:1259] Loaded cluster ID: 63dda9467b6f4644a36c82b2bd8fad32
I20260925 22:00:22.657544 24804 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d: loading cluster ID for follower catalog manager: success
I20260925 22:00:22.658378 24804 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d: acquiring CA information for follower catalog manager: success
I20260925 22:00:22.658664 24804 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 757c2d411fd743cd846bcbde378d295d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:00:22.753222 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:22.750796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.2.70.253:43055)
0925 22:00:22.750959 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:22.750961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:22.750971 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:00:22.750996 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:22.750997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:22.751046 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:22.751117 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:00:22.751267 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:22.751787 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:22.752559 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:22.752763 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:22.752818 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:22.752924 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:00:22.753039 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:00:22.753100 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:22.753114 (+ 14us) server_negotiation.cc:299] Negotiation successful
0925 22:00:22.753132 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
I20260925 22:00:22.757460 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:22.754906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.2.70.252:43575)
0925 22:00:22.755046 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:22.755049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:22.755058 (+ 9us) server_negotiation.cc:407] Connection header received
0925 22:00:22.755101 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:22.755103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:22.755148 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:22.755220 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:00:22.755385 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:22.755954 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:22.756728 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:22.756988 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:22.757046 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:22.757142 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:00:22.757243 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:00:22.757296 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:22.757374 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 22:00:22.757394 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":41,"threads_started":1}
I20260925 22:00:22.761642 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:22.759157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.2.70.253:43055)
0925 22:00:22.759205 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:22.759207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:22.759274 (+ 67us) server_negotiation.cc:407] Connection header received
0925 22:00:22.759322 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:22.759323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:22.759334 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:22.759382 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:00:22.759542 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:22.760082 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:22.760942 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:22.761154 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:22.761211 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:22.761373 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:00:22.761420 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:00:22.761473 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:22.761537 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 22:00:22.761562 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 22:00:22.761881 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:22.759057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.2.70.252:43575)
0925 22:00:22.759205 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:22.759207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:22.759295 (+ 88us) server_negotiation.cc:407] Connection header received
0925 22:00:22.759365 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:22.759366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:22.759374 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:22.759417 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:00:22.759573 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:22.760121 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:22.761186 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:22.761423 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:22.761478 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:22.761650 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:00:22.761679 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:00:22.761704 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:22.761803 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 22:00:22.761823 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260925 22:00:22.762861 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:22.759218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.2.70.254:38349)
0925 22:00:22.759321 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:22.759322 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:22.759329 (+ 7us) server_negotiation.cc:407] Connection header received
0925 22:00:22.759369 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:22.759369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:22.759452 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:22.759533 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:00:22.760291 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:22.761046 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:22.762014 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:22.762285 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:22.762341 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:22.762553 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:00:22.762642 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:00:22.762700 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:22.762747 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 22:00:22.762768 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
I20260925 22:00:22.768476 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 24525
I20260925 22:00:22.773921 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 24600
I20260925 22:00:22.778132 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 24675
2026-09-25T22:00:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2729 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:00:22 dist-test-slave-2g0f krb5kdc[24894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:00:22 dist-test-slave-2g0f krb5kdc[24894](info): set up 2 sockets
Sep 25 22:00:22 dist-test-slave-2g0f krb5kdc[24894](info): commencing operation
krb5kdc: starting...
W20260925 22:00:23.735878 2331 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/test-user.keytab.
Sep 25 22:00:23 dist-test-slave-2g0f krb5kdc[24894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
Entry for principal kudu/127.2.70.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab.
I20260925 22:00:23.856559 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:34719
--webserver_interface=127.2.70.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:34719,127.2.70.253:36451,127.2.70.252:43185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:00:23.937287 24907 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:00:23.937503 24907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:23.937525 24907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:23.937544 24907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:23.938699 24907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:23.938740 24907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:23.938753 24907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:23.938764 24907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:23.938776 24907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:23.940295 24907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:34719,127.2.70.253:36451,127.2.70.252:43185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.254.keytab
--principal=kudu/127.2.70.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.254:34719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:23.940518 24907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:23.940737 24907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:23.945356 24915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:23.945407 24912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:23.945490 24907 server_base.cc:1034] running on GCE node
W20260925 22:00:23.945377 24913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:23.945801 24907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:23.946079 24907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:23.947242 24907 hybrid_clock.cc:648] HybridClock initialized: now 1790373623947227 us; error 27 us; skew 500 ppm
Sep 25 22:00:23 dist-test-slave-2g0f krb5kdc[24894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373623, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:23.949713 24907 init.cc:377] Logged in from keytab as kudu/127.2.70.254@KRBTEST.COM (short username kudu)
I20260925 22:00:23.950116 24907 webserver.cc:460] Webserver started at http://127.2.70.254:45455/ using document root <none> and password file <none>
I20260925 22:00:23.950336 24907 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:23.950379 24907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:23.950492 24907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:23.951603 24907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/data/instance:
uuid: "ab7395d226a4439684cab88b95a875b6"
format_stamp: "Formatted at 2026-09-25 22:00:23 on dist-test-slave-2g0f"
server_key: "c1a752070c41704625cf931b996e49e6"
server_key_iv: "ed81657c0185c31edfb0b9ee0627b169"
server_key_version: "encryptionkey@0"
I20260925 22:00:23.951965 24907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/wal/instance:
uuid: "ab7395d226a4439684cab88b95a875b6"
format_stamp: "Formatted at 2026-09-25 22:00:23 on dist-test-slave-2g0f"
server_key: "c1a752070c41704625cf931b996e49e6"
server_key_iv: "ed81657c0185c31edfb0b9ee0627b169"
server_key_version: "encryptionkey@0"
I20260925 22:00:23.953250 24907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:00:23.953949 24922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:23.954121 24907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 22:00:23.954202 24907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/wal
uuid: "ab7395d226a4439684cab88b95a875b6"
format_stamp: "Formatted at 2026-09-25 22:00:23 on dist-test-slave-2g0f"
server_key: "c1a752070c41704625cf931b996e49e6"
server_key_iv: "ed81657c0185c31edfb0b9ee0627b169"
server_key_version: "encryptionkey@0"
I20260925 22:00:23.954257 24907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:23.966867 24907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:23.968983 24907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:23.969091 24907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:23.972741 24907 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.254:34719
I20260925 22:00:23.972781 24974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.254:34719 every 8 connection(s)
I20260925 22:00:23.973107 24907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/data/info.pb
I20260925 22:00:23.974159 24975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:23.975718 24975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:23.981479 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 24907
I20260925 22:00:23.981557 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-0/wal/instance
Sep 25 22:00:23 dist-test-slave-2g0f krb5kdc[24894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373623, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 22:00:23.981712 2331 external_mini_cluster.cc:1428] Setting key eb8d782d266b5a6c0fe5b931b34463cc
I20260925 22:00:23.985349 24978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:23.976488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.2.70.254:34719)
0925 22:00:23.976640 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:23.976648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:23.976743 (+ 95us) server_negotiation.cc:407] Connection header received
0925 22:00:23.977235 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:23.977237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:23.977254 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:23.977329 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:23.977781 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:23.978623 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:23.979542 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:23.979876 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:23.979972 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:23.982225 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:23.982238 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:23.982238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:23.982245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:23.983823 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:23.984232 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:23.984234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:23.984235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:23.984308 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:23.984489 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:23.984489 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:23.984490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:23.984699 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:23.984779 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:23.985029 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 22:00:23.985153 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260925 22:00:23.985357 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:23.976467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34719 (local address 127.0.0.1:51326)
0925 22:00:23.976624 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:23.976662 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:23.977125 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:23.977334 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:23.977336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:23.977567 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:23.977717 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:23.977721 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:23.978708 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:23.978709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:23.979458 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:23.979460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:23.980007 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:23.980008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:23.980078 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:23.980578 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:23.980586 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:23.982126 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:23.983916 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:23.983923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:23.983925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:23.984161 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:23.984360 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:23.984361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:23.984361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:23.984438 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:23.984786 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:23.984794 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:23.984894 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 22:00:23.985006 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 22:00:23.985218 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":49,"thread_start_us":73,"threads_started":1}
I20260925 22:00:23.986363 24975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:23.986702 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:23.986582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:36451 (local address 127.0.0.1:55582)
0925 22:00:23.986628 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:23.986677 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 22:00:23.986858 24926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.253:36451: connect: Connection refused (error 111)
I20260925 22:00:23.987389 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:23.987288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:36451 (local address 127.0.0.1:55592)
0925 22:00:23.987337 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:23.987366 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 22:00:23.987517 24975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.253:36451: Network error: Client connection negotiation failed: client connection to 127.2.70.253:36451: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.70.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.253.keytab.
Entry for principal kudu/127.2.70.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.253.keytab.
I20260925 22:00:24.007337 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.253:36451
--webserver_interface=127.2.70.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.253.keytab
--principal=kudu/127.2.70.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:34719,127.2.70.253:36451,127.2.70.252:43185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:00:24.036751 24975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } attempt: 1
I20260925 22:00:24.037324 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.037155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:36451 (local address 127.0.0.1:55606)
0925 22:00:24.037230 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.037285 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 22:00:24.037657 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.037570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:36451 (local address 127.0.0.1:55614)
0925 22:00:24.037611 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.037638 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 22:00:24.037770 24975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.253:36451: Network error: Client connection negotiation failed: client connection to 127.2.70.253:36451: connect: Connection refused (error 111)
W20260925 22:00:24.086797 24982 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:00:24.087080 24982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:24.087101 24982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:24.087121 24982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:24.088366 24982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:24.088408 24982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:24.088421 24982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:24.088433 24982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:24.088443 24982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:24.090027 24982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:34719,127.2.70.253:36451,127.2.70.252:43185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.253.keytab
--principal=kudu/127.2.70.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.253:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.70.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:24.090348 24982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:24.090714 24982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:00:24.095588 24990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:24.095600 24988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:24.095599 24987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:24.095804 24982 server_base.cc:1034] running on GCE node
I20260925 22:00:24.096050 24982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:24.096331 24982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:24.097479 24982 hybrid_clock.cc:648] HybridClock initialized: now 1790373624097471 us; error 32 us; skew 500 ppm
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:24.099864 24982 init.cc:377] Logged in from keytab as kudu/127.2.70.253@KRBTEST.COM (short username kudu)
I20260925 22:00:24.100324 24982 webserver.cc:460] Webserver started at http://127.2.70.253:38735/ using document root <none> and password file <none>
I20260925 22:00:24.100541 24982 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:24.100587 24982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:24.100711 24982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:24.101732 24982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/data/instance:
uuid: "1f89fca299984abc9c71decaa94baed0"
format_stamp: "Formatted at 2026-09-25 22:00:24 on dist-test-slave-2g0f"
server_key: "7e081a11e84f0fa547bf003da3bcdf7e"
server_key_iv: "fab44f852bc67addb4f9149a073d6f3f"
server_key_version: "encryptionkey@0"
I20260925 22:00:24.102053 24982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/wal/instance:
uuid: "1f89fca299984abc9c71decaa94baed0"
format_stamp: "Formatted at 2026-09-25 22:00:24 on dist-test-slave-2g0f"
server_key: "7e081a11e84f0fa547bf003da3bcdf7e"
server_key_iv: "fab44f852bc67addb4f9149a073d6f3f"
server_key_version: "encryptionkey@0"
I20260925 22:00:24.103318 24982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 22:00:24.104067 24997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:24.104223 24982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:00:24.104303 24982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/wal
uuid: "1f89fca299984abc9c71decaa94baed0"
format_stamp: "Formatted at 2026-09-25 22:00:24 on dist-test-slave-2g0f"
server_key: "7e081a11e84f0fa547bf003da3bcdf7e"
server_key_iv: "fab44f852bc67addb4f9149a073d6f3f"
server_key_version: "encryptionkey@0"
I20260925 22:00:24.104369 24982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:24.105929 24975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } attempt: 2
I20260925 22:00:24.106371 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.106229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:36451 (local address 127.0.0.1:55630)
0925 22:00:24.106289 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.106333 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 22:00:24.106688 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.106602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:36451 (local address 127.0.0.1:55634)
0925 22:00:24.106639 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.106666 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.253:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 22:00:24.106770 24975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.253:36451: Network error: Client connection negotiation failed: client connection to 127.2.70.253:36451: connect: Connection refused (error 111)
I20260925 22:00:24.114148 24982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:24.115885 24982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:24.115991 24982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:24.119659 24982 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.253:36451
I20260925 22:00:24.119689 25049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.253:36451 every 8 connection(s)
I20260925 22:00:24.120036 24982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/data/info.pb
I20260925 22:00:24.121002 25050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:24.121994 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 24982
I20260925 22:00:24.122125 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-1/wal/instance
I20260925 22:00:24.122294 2331 external_mini_cluster.cc:1428] Setting key 5422303bc265258f6d952a178996f554
I20260925 22:00:24.122694 25050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.253@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.252@KRBTEST.COM; defaulting to no policy
I20260925 22:00:24.132416 24978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.123382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.2.70.254:34719)
0925 22:00:24.123432 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.123434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.123660 (+ 226us) server_negotiation.cc:407] Connection header received
0925 22:00:24.124393 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.124395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.124413 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.124481 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.125066 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.125811 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.126861 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.127166 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.127243 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.129457 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.129459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.129459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.129468 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.130954 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.131398 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.131399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.131399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.131462 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.131738 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.131739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.131739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.131907 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.131981 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.132261 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.132343 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 22:00:24.132563 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.123374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34719 (local address 127.0.0.1:51330)
0925 22:00:24.123520 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.123564 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:24.124286 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:24.124545 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:24.124549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:24.124787 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:24.124983 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.124987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.125928 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.125929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.126769 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.126771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.127286 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.127288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.127404 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.127986 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:24.127995 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:24.129366 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:24.131035 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.131042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.131044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.131297 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.131543 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.131544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.131544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.131655 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.132012 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:24.132018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:24.132143 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 22:00:24.132234 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 22:00:24.132359 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.252@KRBTEST.COM" created.
I20260925 22:00:24.133201 25050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.253@KRBTEST.COM for kudu/127.2.70.253@KRBTEST.COM
I20260925 22:00:24.141109 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.133459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:36451 (local address 127.0.0.1:55648)
0925 22:00:24.133500 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.133510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:24.133540 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:24.133787 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:24.133788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:24.133923 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:24.133981 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.133983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.135335 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.135335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.135885 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.135887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.136536 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.136537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.136599 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.137093 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:24.137098 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:24.138206 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:24.140190 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.140191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.140192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.140341 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.140539 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.140539 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.140539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.140584 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.140873 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:24.140873 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:24.140908 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 22:00:24.140950 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 22:00:24.141039 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 22:00:24.141237 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.133529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.2.70.253:36451)
0925 22:00:24.133661 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.133665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.133680 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:00:24.133694 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.133696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.133706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.133766 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.134076 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.135231 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.135963 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.136444 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.136529 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.138291 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.138301 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.138301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.138309 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.140097 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.140413 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.140416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.140416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.140479 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.140636 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.140636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.140637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.140811 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.140897 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.140997 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.141134 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260925 22:00:24.141781 25050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:24.142113 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.142004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43185 (local address 127.0.0.1:54692)
0925 22:00:24.142050 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.142095 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 22:00:24.142242 24999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
I20260925 22:00:24.142657 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.142567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43185 (local address 127.0.0.1:54696)
0925 22:00:24.142609 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.142635 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 22:00:24.142764 25050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:43185: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.252.keytab.
Entry for principal kudu/127.2.70.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.252.keytab.
I20260925 22:00:24.148840 2331 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdR78Ha/build/release/bin/kudu
/tmp/dist-test-taskdR78Ha/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.252:43185
--webserver_interface=127.2.70.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.252.keytab
--principal=kudu/127.2.70.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.212:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.254:34719,127.2.70.253:36451,127.2.70.252:43185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:00:24.191937 25050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } attempt: 1
I20260925 22:00:24.192421 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.192299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43185 (local address 127.0.0.1:54704)
0925 22:00:24.192359 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.192391 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 22:00:24.192726 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.192649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43185 (local address 127.0.0.1:54718)
0925 22:00:24.192683 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.192707 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 22:00:24.192817 25050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:43185: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
I20260925 22:00:24.197921 24975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } attempt: 3
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373623, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for kudu/127.2.70.253@KRBTEST.COM
I20260925 22:00:24.204789 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.198346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.2.70.253:36451)
0925 22:00:24.198430 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.198432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.198446 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:00:24.198577 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.198578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.198592 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.198642 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.198962 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.199476 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.200177 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.200393 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.200450 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.202408 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.202410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.202410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.202431 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.203713 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.204108 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.204109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.204109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.204176 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.204358 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.204359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.204359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.204475 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.204523 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.204661 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.204724 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:00:24.204835 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.198254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:36451 (local address 127.0.0.1:55656)
0925 22:00:24.198320 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.198330 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:24.198463 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:24.198653 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:24.198654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:24.198811 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:24.198883 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.198885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.199557 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.199558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.200108 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.200110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.200454 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.200455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.200531 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.200985 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:24.200990 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:24.202300 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:24.203789 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.203791 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.203791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.204026 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.204241 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.204242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.204242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.204306 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.204525 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:24.204526 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:24.204577 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 22:00:24.204642 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 22:00:24.204742 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 22:00:24.205282 24975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:24.205610 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.205510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43185 (local address 127.0.0.1:54728)
0925 22:00:24.205550 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.205583 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 22:00:24.205948 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.205874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43185 (local address 127.0.0.1:54740)
0925 22:00:24.205906 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.205931 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 22:00:24.206099 24975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:43185: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
W20260925 22:00:24.229970 25057 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:00:24.230190 25057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:00:24.230207 25057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:00:24.230226 25057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:00:24.231383 25057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:00:24.231426 25057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:00:24.231438 25057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:00:24.231448 25057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:00:24.231459 25057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:00:24.232965 25057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.212:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.254:34719,127.2.70.253:36451,127.2.70.252:43185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/krb5kdc/kudu_127.2.70.252.keytab
--principal=kudu/127.2.70.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.252:43185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.70.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 20:55:39 UTC on ccycloud.quasar-hwtoeh.root.comops.site
build id 4673252
I20260925 22:00:24.233193 25057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:00:24.233409 25057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:00:24.237264 24975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } attempt: 1
I20260925 22:00:24.237689 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.237515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43185 (local address 127.0.0.1:54746)
0925 22:00:24.237618 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.237658 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:00:24.238047 25065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:24.238150 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.238031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43185 (local address 127.0.0.1:54754)
0925 22:00:24.238089 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.238124 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:00:24.238175 25062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:00:24.238236 24975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.252:43185: Network error: Client connection negotiation failed: client connection to 127.2.70.252:43185: connect: Connection refused (error 111)
W20260925 22:00:24.238047 25063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:00:24.238062 25057 server_base.cc:1034] running on GCE node
I20260925 22:00:24.238472 25057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:00:24.238719 25057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:00:24.239928 25057 hybrid_clock.cc:648] HybridClock initialized: now 1790373624239914 us; error 27 us; skew 500 ppm
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:00:24.242369 25057 init.cc:377] Logged in from keytab as kudu/127.2.70.252@KRBTEST.COM (short username kudu)
I20260925 22:00:24.242791 25057 webserver.cc:460] Webserver started at http://127.2.70.252:35211/ using document root <none> and password file <none>
I20260925 22:00:24.243010 25057 fs_manager.cc:359] Metadata directory not provided
I20260925 22:00:24.243055 25057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:00:24.243148 25057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:00:24.244210 25057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/data/instance:
uuid: "5d53d20613ee4db5adb567b20b492035"
format_stamp: "Formatted at 2026-09-25 22:00:24 on dist-test-slave-2g0f"
server_key: "fc08aea2982400ff82981a5ec749826e"
server_key_iv: "2c15627fc0fc11550cedf8da2ef7de13"
server_key_version: "encryptionkey@0"
I20260925 22:00:24.244493 25057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/wal/instance:
uuid: "5d53d20613ee4db5adb567b20b492035"
format_stamp: "Formatted at 2026-09-25 22:00:24 on dist-test-slave-2g0f"
server_key: "fc08aea2982400ff82981a5ec749826e"
server_key_iv: "2c15627fc0fc11550cedf8da2ef7de13"
server_key_version: "encryptionkey@0"
I20260925 22:00:24.245661 25057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 22:00:24.246381 25072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:00:24.246560 25057 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 22:00:24.246639 25057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/data,/tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/wal
uuid: "5d53d20613ee4db5adb567b20b492035"
format_stamp: "Formatted at 2026-09-25 22:00:24 on dist-test-slave-2g0f"
server_key: "fc08aea2982400ff82981a5ec749826e"
server_key_iv: "2c15627fc0fc11550cedf8da2ef7de13"
server_key_version: "encryptionkey@0"
I20260925 22:00:24.246701 25057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:00:24.255069 25057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:00:24.257179 25057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:00:24.257313 25057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:00:24.260964 25050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } attempt: 2
I20260925 22:00:24.261046 25057 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.252:43185
I20260925 22:00:24.261086 25124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.252:43185 every 8 connection(s)
I20260925 22:00:24.261472 25057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/data/info.pb
I20260925 22:00:24.262151 25125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:00:24.263278 2331 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdR78Ha/build/release/bin/kudu as pid 25057
I20260925 22:00:24.263384 2331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdR78Ha/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790373357410105-2331-0/minicluster-data/master-2/wal/instance
I20260925 22:00:24.263582 2331 external_mini_cluster.cc:1428] Setting key d6228488b20e2ad5a8b23074ed63a844
I20260925 22:00:24.264191 25125 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.253@KRBTEST.COM for kudu/127.2.70.252@KRBTEST.COM
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.252@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.252@KRBTEST.COM
I20260925 22:00:24.271282 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.261288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43185 (local address 127.0.0.1:54766)
0925 22:00:24.261347 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.261360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:24.261418 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:24.262843 (+ 1425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:24.262844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:24.262979 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:24.263060 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.263062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.264731 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.264732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.265469 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.265472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.266228 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.266229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.266288 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.266753 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:24.266758 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:24.268131 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:24.269984 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.269985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.269986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.270198 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.270445 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.270446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.270447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.270511 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.270944 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:24.270945 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:24.271001 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 22:00:24.271076 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:00:24.271178 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 22:00:24.271492 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.261442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.2.70.252:43185)
0925 22:00:24.262012 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.262021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.262052 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:00:24.262626 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.262635 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.262698 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.262846 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.263535 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.264641 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.265537 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.266149 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.266251 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.268206 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.268214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.268214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.268220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.269908 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.270275 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.270277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.270278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.270376 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.270578 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.270579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.270580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.270878 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.270959 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.271154 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.271281 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":529,"thread_start_us":479,"threads_started":1}
I20260925 22:00:24.273001 25050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0: Bootstrap starting.
I20260925 22:00:24.273245 24978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.264816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.2.70.254:34719)
0925 22:00:24.264860 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.264862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.265042 (+ 180us) server_negotiation.cc:407] Connection header received
0925 22:00:24.265100 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.265101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.265112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.265163 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.265726 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.266175 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.267149 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.267435 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.267497 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.270110 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.270111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.270112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.270119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.272000 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.272443 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.272445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.272445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.272499 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.272699 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.272700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.272700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.272833 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.272873 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.273079 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.273153 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 22:00:24.273259 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.264764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.254:34719 (local address 127.0.0.1:51334)
0925 22:00:24.264931 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.264972 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:24.265040 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:24.265184 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:24.265186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:24.265527 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:24.265654 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.265656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.266259 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.266260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.267042 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.267045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.267548 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.267550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.267642 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.268122 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:24.268131 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:24.270017 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:24.272103 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.272110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.272112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.272356 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.272568 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.272568 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.272568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.272623 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.272917 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:24.272923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:24.272972 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 22:00:24.273035 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 22:00:24.273152 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":58,"thread_start_us":71,"threads_started":1}
I20260925 22:00:24.273960 25050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:24.273950 25125 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:24.274259 25050 log.cc:824] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:24.274405 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.263922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54768 (local address 127.2.70.252:43185)
0925 22:00:24.264667 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.264669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.264677 (+ 8us) server_negotiation.cc:407] Connection header received
0925 22:00:24.264695 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.264695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.264704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.264762 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.265462 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.266240 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.267071 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.267374 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.267429 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.270352 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.270353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.270353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.270358 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.272831 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.273221 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.273222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.273222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.273287 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.273477 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.273478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.273479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.273605 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.273729 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.274235 (+ 506us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.274315 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":720,"thread_start_us":76,"threads_started":1}
I20260925 22:00:24.275256 25050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0: No bootstrap required, opened a new log
I20260925 22:00:24.277004 25050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } }
I20260925 22:00:24.277139 25050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f89fca299984abc9c71decaa94baed0, State: Initialized, Role: FOLLOWER
I20260925 22:00:24.277215 25050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } }
I20260925 22:00:24.277562 25050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.252@KRBTEST.COM for kudu/127.2.70.253@KRBTEST.COM
I20260925 22:00:24.279294 25132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } } }
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.254@KRBTEST.COM
I20260925 22:00:24.279428 25132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 22:00:24.279623 25143 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:00:24.279662 25143 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:00:24.281818 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.274237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.2.70.253:36451)
0925 22:00:24.274519 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.274522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.274530 (+ 8us) server_negotiation.cc:407] Connection header received
0925 22:00:24.274548 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.274549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.274558 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.274612 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.274893 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.275549 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.276495 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.276776 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.276847 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.278825 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.278826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.278826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.278833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.280635 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.281003 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.281004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.281004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.281063 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.281259 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.281260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.281260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.281417 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.281457 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.281658 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.281738 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260925 22:00:24.282292 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.274184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.253:36451 (local address 127.0.0.1:55672)
0925 22:00:24.274213 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.274223 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:24.274256 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:24.274642 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:24.274643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:24.274749 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:24.274796 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.274797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.275627 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.275628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.276414 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.276416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.276852 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.276853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.276924 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.277353 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:24.277358 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:24.278728 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:24.280721 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.280722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.280723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.280922 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.281136 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.281137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.281137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.281195 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.281501 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:24.281502 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:24.281557 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 22:00:24.282096 (+ 539us) client_negotiation.cc:240] Negotiation successful
0925 22:00:24.282203 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 22:00:24.279858 24982 master_runner.cc:186] Error getting master registration for 127.2.70.254:34719: OK, instance_id { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" instance_seqno: 1790373623971670 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:00:24.282692 24982 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:00:24.283054 25125 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:00:24.283147 24978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.275155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.2.70.254:34719)
0925 22:00:24.275186 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.275187 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.275194 (+ 7us) server_negotiation.cc:407] Connection header received
0925 22:00:24.275231 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.275232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.275240 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.275283 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.275724 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.276456 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.277196 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.277457 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.277516 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.280222 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.280223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.280223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.280230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.281734 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.282116 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.282117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.282118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.282173 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.282347 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.282348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.282348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.282463 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.282594 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.282827 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.283070 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.252@KRBTEST.COM for kudu/127.2.70.252@KRBTEST.COM
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.253@KRBTEST.COM
I20260925 22:00:24.290660 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.283286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43185 (local address 127.0.0.1:54782)
0925 22:00:24.283373 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.283384 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:24.283424 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:24.283612 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:24.283613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:24.283750 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:24.283786 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.283788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.284578 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.284578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.285020 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.285022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.285487 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.285487 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.285532 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.286151 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:24.286156 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:24.287483 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:24.289079 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.289080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.289081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.289293 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.289960 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.289961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.289961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.290014 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.290323 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:24.290324 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:24.290363 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 22:00:24.290427 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:00:24.290526 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 22:00:24.290747 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.283347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.2.70.252:43185)
0925 22:00:24.283486 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.283488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.283495 (+ 7us) server_negotiation.cc:407] Connection header received
0925 22:00:24.283513 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.283513 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.283523 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.283575 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.283877 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.284513 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.285086 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.285391 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.285445 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.287552 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.287554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.287554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.287560 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.288997 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.289853 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.289854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.289854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.289898 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.290065 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.290066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.290066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.290264 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.290322 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.290471 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.290593 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":54}
I20260925 22:00:24.291452 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.283621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.2.70.253:36451)
0925 22:00:24.283653 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.283655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.283661 (+ 6us) server_negotiation.cc:407] Connection header received
0925 22:00:24.283679 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.283680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.283688 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.283741 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.284278 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.284971 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.285836 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.286027 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.286078 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.288477 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.288479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.288479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.288486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.289782 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.290122 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.290123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.290123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.290170 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.290384 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.290386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.290386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.290511 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.290700 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.291223 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.291295 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 22:00:24.292325 25125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035: Bootstrap starting.
I20260925 22:00:24.292737 2331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:00:24.293031 25125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:24.293319 25125 log.cc:824] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:24.294170 25125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035: No bootstrap required, opened a new log
I20260925 22:00:24.293535 24949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ab7395d226a4439684cab88b95a875b6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51340
I20260925 22:00:24.296172 25125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } }
I20260925 22:00:24.296334 25125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d53d20613ee4db5adb567b20b492035, State: Initialized, Role: FOLLOWER
I20260925 22:00:24.296418 25125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } }
I20260925 22:00:24.296759 25125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:24.296720 25146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } } }
I20260925 22:00:24.296814 25146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 22:00:24.298903 25157 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:00:24.298946 25157 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:00:24.298780 25057 master_runner.cc:186] Error getting master registration for 127.2.70.254:34719: OK, instance_id { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" instance_seqno: 1790373623971670 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:00:24.300493 25057 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:00:24.313369 24975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } attempt: 2
Sep 25 22:00:24 dist-test-slave-2g0f krb5kdc[24894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790373623, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.254@KRBTEST.COM for kudu/127.2.70.252@KRBTEST.COM
I20260925 22:00:24.320540 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.313733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.252:43185 (local address 127.0.0.1:54784)
0925 22:00:24.313839 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 22:00:24.313850 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:00:24.313898 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:00:24.314022 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:00:24.314023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:00:24.314211 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:00:24.314263 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.314264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.314858 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.314859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.315669 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:00:24.315671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.316055 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.316056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:00:24.316115 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.316532 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:00:24.316536 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:00:24.317991 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:00:24.319504 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.319506 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.319506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.319716 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.319920 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:00:24.319921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:00:24.319921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:00:24.319980 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:00:24.320235 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:00:24.320236 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:00:24.320280 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 22:00:24.320340 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 22:00:24.320439 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 22:00:24.320540 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.313791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.2.70.252:43185)
0925 22:00:24.313843 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.313845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.313883 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:00:24.313950 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.313951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.313960 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.314009 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.314320 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.314784 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.315750 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.315985 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.316036 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.318076 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.318078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.318078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.318085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.319425 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.319772 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.319774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.319774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.319854 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.320025 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.320042 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.320042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.320178 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.320234 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.320371 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.320444 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 22:00:24.322329 24975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6: Bootstrap starting.
I20260925 22:00:24.323063 24975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6: Neither blocks nor log segments found. Creating new log.
I20260925 22:00:24.323364 24975 log.cc:824] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:00:24.324599 24975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6: No bootstrap required, opened a new log
I20260925 22:00:24.326041 24975 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } }
I20260925 22:00:24.326184 24975 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7395d226a4439684cab88b95a875b6, State: Initialized, Role: FOLLOWER
I20260925 22:00:24.326267 24975 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } }
I20260925 22:00:24.326584 24975 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:00:24.326609 24949 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:00:24.326650 24949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:24.327473 24949 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } }
I20260925 22:00:24.327719 24949 leader_election.cc:288] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [CANDIDATE]: Term 1 election: Requested vote from peers 1f89fca299984abc9c71decaa94baed0 (127.2.70.253:36451), 5d53d20613ee4db5adb567b20b492035 (127.2.70.252:43185)
I20260925 22:00:24.328518 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab7395d226a4439684cab88b95a875b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1f89fca299984abc9c71decaa94baed0"
I20260925 22:00:24.328670 25024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:24.329552 25024 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7395d226a4439684cab88b95a875b6 in term 1.
I20260925 22:00:24.330739 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab7395d226a4439684cab88b95a875b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5d53d20613ee4db5adb567b20b492035"
I20260925 22:00:24.330889 25099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:00:24.331735 25099 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7395d226a4439684cab88b95a875b6 in term 1.
I20260925 22:00:24.332108 24924 leader_election.cc:302] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d53d20613ee4db5adb567b20b492035, ab7395d226a4439684cab88b95a875b6; no voters:
I20260925 22:00:24.326545 25160 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } } }
I20260925 22:00:24.332578 25160 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:00:24.332671 25160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:00:24.332760 25160 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [term 1 LEADER]: Becoming Leader. State: Replica: ab7395d226a4439684cab88b95a875b6, State: Running, Role: LEADER
I20260925 22:00:24.332882 25160 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } }
I20260925 22:00:24.333387 25160 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab7395d226a4439684cab88b95a875b6. Latest consensus state: current_term: 1 leader_uuid: "ab7395d226a4439684cab88b95a875b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } } }
I20260925 22:00:24.333509 25160 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [sys.catalog]: This master's current role is: LEADER
W20260925 22:00:24.328809 25171 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:00:24.333796 25171 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:00:24.334645 24907 master_runner.cc:263] No leader master found from master ab7395d226a4439684cab88b95a875b6
I20260925 22:00:24.334687 24907 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:00:24.335592 25179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:00:24.335863 25179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:00:24.346366 25099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [term 1 FOLLOWER]: Refusing update from remote peer ab7395d226a4439684cab88b95a875b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:00:24.347112 25177 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:00:24.349717 25146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab7395d226a4439684cab88b95a875b6. Latest consensus state: current_term: 1 leader_uuid: "ab7395d226a4439684cab88b95a875b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } } }
I20260925 22:00:24.349809 25146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:00:24.350414 25024 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [term 1 FOLLOWER]: Refusing update from remote peer ab7395d226a4439684cab88b95a875b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:00:24.350786 25177 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:00:24.352126 25132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab7395d226a4439684cab88b95a875b6. Latest consensus state: current_term: 1 leader_uuid: "ab7395d226a4439684cab88b95a875b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } } }
I20260925 22:00:24.352219 25132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:00:24.353650 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.335273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.2.70.252:43185)
0925 22:00:24.335311 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.335312 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.340592 (+ 5280us) server_negotiation.cc:407] Connection header received
0925 22:00:24.340682 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.340683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.340694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.340721 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.341361 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.342100 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.343108 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.343408 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.343445 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.345626 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.345628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.345628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.345636 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.348545 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.352799 (+ 4254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.352800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.352800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.352863 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.353074 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.353075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.353075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.353207 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.353280 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.353479 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.353558 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 22:00:24.355167 24978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.334168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.2.70.254:34719)
0925 22:00:24.334231 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.334233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.339992 (+ 5759us) server_negotiation.cc:407] Connection header received
0925 22:00:24.340108 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.340109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.340119 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.340147 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.343037 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.343737 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.349039 (+ 5302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.349356 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.349396 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.352473 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.352474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.352474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.352481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.354020 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.354385 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.354386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.354386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.354436 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.354616 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.354617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.354617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.354739 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.354810 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.354987 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.355077 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:00:24.355801 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:24.335048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55694 (local address 127.2.70.253:36451)
0925 22:00:24.335100 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.335102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.340051 (+ 4949us) server_negotiation.cc:407] Connection header received
0925 22:00:24.340163 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.340164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.340175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.340203 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.341229 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.341967 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.344757 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:24.345066 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:24.345106 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:24.350876 (+ 5770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:00:24.350877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:00:24.350877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:00:24.350885 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:00:24.352709 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.354814 (+ 2105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.354815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.354816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.354871 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:00:24.355047 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:00:24.355048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:00:24.355048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:00:24.355182 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:00:24.355254 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:24.355627 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 22:00:24.355709 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 22:00:24.358162 25177 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab7395d226a4439684cab88b95a875b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } } }
I20260925 22:00:24.358253 25177 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:24.358712 25132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab7395d226a4439684cab88b95a875b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } } }
I20260925 22:00:24.358788 25132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:00:24.359099 25183 mvcc.cc:205] Tried to move back new op lower bound from 7333370365318328320 to 7333370365268561920. Current Snapshot: MvccSnapshot[applied={T|T < 7333370365318328320}]
I20260925 22:00:24.359453 25186 mvcc.cc:205] Tried to move back new op lower bound from 7333370365318328320 to 7333370365268561920. Current Snapshot: MvccSnapshot[applied={T|T < 7333370365318328320}]
I20260925 22:00:24.359508 25160 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab7395d226a4439684cab88b95a875b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } } }
I20260925 22:00:24.359604 25160 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6 [sys.catalog]: This master's current role is: LEADER
I20260925 22:00:24.359969 25146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab7395d226a4439684cab88b95a875b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7395d226a4439684cab88b95a875b6" member_type: VOTER last_known_addr { host: "127.2.70.254" port: 34719 } } peers { permanent_uuid: "1f89fca299984abc9c71decaa94baed0" member_type: VOTER last_known_addr { host: "127.2.70.253" port: 36451 } } peers { permanent_uuid: "5d53d20613ee4db5adb567b20b492035" member_type: VOTER last_known_addr { host: "127.2.70.252" port: 43185 } } }
I20260925 22:00:24.360073 25146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:00:24.360054 25184 mvcc.cc:205] Tried to move back new op lower bound from 7333370365318328320 to 7333370365268561920. Current Snapshot: MvccSnapshot[applied={T|T < 7333370365318328320}]
I20260925 22:00:24.365012 25179 catalog_manager.cc:1347] Generated new cluster ID: b7e8b01a9b0c4b97b5cd2337edd5fe96
I20260925 22:00:24.365118 25179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:00:24.414539 25179 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:00:24.415347 25179 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:00:24.423243 25179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab7395d226a4439684cab88b95a875b6: Generated new TSK 0
I20260925 22:00:24.423473 25179 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:00:24.451902 24978 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.450157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51356 (local address 127.2.70.254:34719)
0925 22:00:24.450201 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.450203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.451211 (+ 1008us) server_negotiation.cc:407] Connection header received
0925 22:00:24.451315 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.451316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.451328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.451357 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.451803 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51356: BlockingRecv error: recv got EOF from 127.0.0.1:51356 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 22:00:24.452673 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.450518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.2.70.253:36451)
0925 22:00:24.450557 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.450559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.452124 (+ 1565us) server_negotiation.cc:407] Connection header received
0925 22:00:24.452219 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.452220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.452231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.452258 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.452608 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55704: BlockingRecv error: recv got EOF from 127.0.0.1:55704 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 22:00:24.453392 25128 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.450898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54796 (local address 127.2.70.252:43185)
0925 22:00:24.450935 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.450937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.452913 (+ 1976us) server_negotiation.cc:407] Connection header received
0925 22:00:24.452998 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.452999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.453020 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.453047 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.453335 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54796: BlockingRecv error: recv got EOF from 127.0.0.1:54796 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 22:00:24.455761 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.455052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.2.70.253:36451)
0925 22:00:24.455088 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.455089 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.455277 (+ 188us) server_negotiation.cc:407] Connection header received
0925 22:00:24.455376 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.455377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.455395 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.455424 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.455713 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55706: BlockingRecv error: recv got EOF from 127.0.0.1:55706 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 22:00:24.458793 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.458055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.2.70.253:36451)
0925 22:00:24.458089 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.458090 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.458327 (+ 237us) server_negotiation.cc:407] Connection header received
0925 22:00:24.458416 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.458417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.458427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.458454 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.458744 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55712: BlockingRecv error: recv got EOF from 127.0.0.1:55712 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 22:00:24.462415 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.461791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.2.70.253:36451)
0925 22:00:24.461836 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.461838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.461959 (+ 121us) server_negotiation.cc:407] Connection header received
0925 22:00:24.462057 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.462058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.462069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.462094 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.462367 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55718: BlockingRecv error: recv got EOF from 127.0.0.1:55718 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 22:00:24.468516 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.467583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.2.70.253:36451)
0925 22:00:24.467624 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.467626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.467942 (+ 316us) server_negotiation.cc:407] Connection header received
0925 22:00:24.468057 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.468058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.468071 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.468100 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.468456 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55720: BlockingRecv error: recv got EOF from 127.0.0.1:55720 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 22:00:24.478350 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.477673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.2.70.253:36451)
0925 22:00:24.477717 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.477718 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.477824 (+ 106us) server_negotiation.cc:407] Connection header received
0925 22:00:24.477910 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.477911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.477923 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.477967 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.478286 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55724: BlockingRecv error: recv got EOF from 127.0.0.1:55724 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 22:00:24.496037 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.495378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.2.70.253:36451)
0925 22:00:24.495424 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.495426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.495496 (+ 70us) server_negotiation.cc:407] Connection header received
0925 22:00:24.495569 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.495570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.495579 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.495628 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.495974 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55732: BlockingRecv error: recv got EOF from 127.0.0.1:55732 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 22:00:24.529942 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.529261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.2.70.253:36451)
0925 22:00:24.529353 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.529355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.529428 (+ 73us) server_negotiation.cc:407] Connection header received
0925 22:00:24.529494 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.529495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.529508 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.529558 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.529876 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55734: BlockingRecv error: recv got EOF from 127.0.0.1:55734 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 22:00:24.595769 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.595185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.2.70.253:36451)
0925 22:00:24.595257 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.595259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.595305 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:00:24.595376 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.595377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.595388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.595432 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.595723 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55738: BlockingRecv error: recv got EOF from 127.0.0.1:55738 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 22:00:24.725636 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.724976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.2.70.253:36451)
0925 22:00:24.725054 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.725056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.725117 (+ 61us) server_negotiation.cc:407] Connection header received
0925 22:00:24.725189 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.725190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.725202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.725252 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.725576 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55744: BlockingRecv error: recv got EOF from 127.0.0.1:55744 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 22:00:24.983476 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:00:24.982821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.2.70.253:36451)
0925 22:00:24.982892 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:24.982894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:24.982903 (+ 9us) server_negotiation.cc:407] Connection header received
0925 22:00:24.983009 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:24.983010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:24.983020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:24.983076 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:00:24.983416 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55746: BlockingRecv error: recv got EOF from 127.0.0.1:55746 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:00:25.280170 25143 catalog_manager.cc:1259] Loaded cluster ID: b7e8b01a9b0c4b97b5cd2337edd5fe96
I20260925 22:00:25.280252 25143 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0: loading cluster ID for follower catalog manager: success
I20260925 22:00:25.281054 25143 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0: acquiring CA information for follower catalog manager: success
I20260925 22:00:25.281347 25143 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f89fca299984abc9c71decaa94baed0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:00:25.299362 25157 catalog_manager.cc:1259] Loaded cluster ID: b7e8b01a9b0c4b97b5cd2337edd5fe96
I20260925 22:00:25.299436 25157 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035: loading cluster ID for follower catalog manager: success
I20260925 22:00:25.300261 25157 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035: acquiring CA information for follower catalog manager: success
I20260925 22:00:25.300551 25157 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d53d20613ee4db5adb567b20b492035: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:00:25.500208 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:25.496782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.2.70.253:36451)
0925 22:00:25.496910 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:25.496912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:25.496928 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:00:25.497052 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:25.497053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:25.497106 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:25.497208 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:00:25.497337 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:25.497857 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:25.499040 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:25.499380 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:25.499443 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:25.499788 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:00:25.499965 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:00:25.500068 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:25.500085 (+ 17us) server_negotiation.cc:299] Negotiation successful
0925 22:00:25.500114 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
I20260925 22:00:25.508203 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:25.504860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.2.70.252:43185)
0925 22:00:25.505025 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:25.505026 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:25.505036 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:00:25.505132 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:25.505133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:25.505187 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:25.505349 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:00:25.505469 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:25.506255 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:25.507264 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:25.507520 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:25.507582 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:25.507781 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:00:25.507954 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:00:25.508046 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:25.508082 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 22:00:25.508109 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260925 22:00:25.513672 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:25.510116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.2.70.254:34719)
0925 22:00:25.510423 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:25.510426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:25.510434 (+ 8us) server_negotiation.cc:407] Connection header received
0925 22:00:25.510480 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:25.510481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:25.510548 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:25.510642 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:00:25.510773 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:25.511501 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:25.512744 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:25.513047 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:25.513112 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:25.513311 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:00:25.513418 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:00:25.513484 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:25.513555 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:00:25.513577 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":55,"threads_started":1}
I20260925 22:00:25.514436 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:25.510215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54816 (local address 127.2.70.252:43185)
0925 22:00:25.510276 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:25.510277 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:25.510469 (+ 192us) server_negotiation.cc:407] Connection header received
0925 22:00:25.510533 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:25.510533 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:25.510543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:25.510592 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:00:25.510730 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:25.511501 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:25.513536 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:25.513792 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:25.513833 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:25.514013 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:00:25.514045 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:00:25.514069 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:25.514327 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 22:00:25.514353 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:00:25.515473 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:00:25.510314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.2.70.253:36451)
0925 22:00:25.510354 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 22:00:25.510356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 22:00:25.510755 (+ 399us) server_negotiation.cc:407] Connection header received
0925 22:00:25.511685 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:00:25.511686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:00:25.511700 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:00:25.511727 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:00:25.511952 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:25.512678 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:25.514597 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:00:25.514872 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:00:25.514912 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:00:25.515209 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:00:25.515245 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:00:25.515298 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:00:25.515355 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 22:00:25.515381 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 22:00:25.516033 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 24907
I20260925 22:00:25.521128 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 24982
I20260925 22:00:25.526829 2331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdR78Ha/build/release/bin/kudu with pid 25057
2026-09-25T22:00:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2751 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5481 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (268124 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST